You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@rocketmq.apache.org by GitBox <gi...@apache.org> on 2022/04/22 03:47:27 UTC
[GitHub] [rocketmq] hzh0425 opened a new pull request, #4195: Feature/ha controller
hzh0425 opened a new pull request, #4195:
URL: https://github.com/apache/rocketmq/pull/4195
## What is the purpose of the change
Add a controller for ha-service, the controller is based on dledger.
## Brief changelog
- Add a controller, based on dledger
- Add some options in name-srv config
- Add controller into name-srv, add controllerRequestProcessor.
## Verifying this change
XXXX
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, `it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR`.
- [x] Make sure there is a [Github issue](https://github.com/apache/rocketmq/issues) filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
- [x] Format the pull request title like `[ISSUE #123] Fix UnknownException when host config not exist`. Each commit in the pull request should have a meaningful subject line and body.
- [x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
- [x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/apache/rocketmq/tree/master/test).
- [x] Run `mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle` to make sure basic checks pass. Run `mvn clean install -DskipITs` to make sure unit-test pass. Run `mvn clean test-compile failsafe:integration-test` to make sure integration-test pass.
- [ ] If this contribution is large, please file an [Apache Individual Contributor License Agreement](http://www.apache.org/licenses/#clas).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857055538
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
+ } else {
+ brokerId = brokerIdTable.get(brokerAddress);
+ }
+ response.setBrokerId(brokerId);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch());
+
+ if (replicasInfo.isMasterAlive()) {
+ // If the master is alive, just return master info.
+ response.setMasterAddress(replicasInfo.getMasterAddress());
+ return result;
+ } else {
+ // If the master is not alive, we should elect a new master:
+ // Case1: This replicas was in sync state set list
+ // Case2: The option {EnableElectUncleanMaster} is true
+ canBeElectedAsMaster = replicasInfo.getSyncStateSet().contains(brokerAddress) || this.enableElectUncleanMaster;
Review Comment:
实际上这个时候 master 不存在, 情况应该是 syncStateSet 中已经没有可选的 broker 了, 所以上线了一个 replicas 可以直接根据 enableElectUncleanMaster 来选.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dugenkui03 commented on a diff in pull request #4195: Dledger controller
Posted by GitBox <gi...@apache.org>.
dugenkui03 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r855767880
##########
common/src/main/java/org/apache/rocketmq/common/namesrv/NamesrvConfig.java:
##########
@@ -200,4 +221,68 @@ public boolean isEnableAllTopicList() {
public void setEnableAllTopicList(boolean enableAllTopicList) {
this.enableAllTopicList = enableAllTopicList;
}
+
+ public boolean isStartupController() {
+ return isStartupController;
+ }
+
+ public void setStartupController(boolean startupController) {
+ isStartupController = startupController;
+ }
+
+ public int getControllerThreadPoolNums() {
+ return controllerThreadPoolNums;
+ }
+
+ public void setControllerThreadPoolNums(int controllerThreadPoolNums) {
+ this.controllerThreadPoolNums = controllerThreadPoolNums;
+ }
+
+ public int getControllerRequestThreadPoolQueueCapacity() {
+ return controllerRequestThreadPoolQueueCapacity;
+ }
+
+ public void setControllerRequestThreadPoolQueueCapacity(int controllerRequestThreadPoolQueueCapacity) {
+ this.controllerRequestThreadPoolQueueCapacity = controllerRequestThreadPoolQueueCapacity;
+ }
+
+ public String getControllerDLegerGroup() {
+ return controllerDLegerGroup;
+ }
+
+ public void setControllerDLegerGroup(String controllerDLegerGroup) {
+ this.controllerDLegerGroup = controllerDLegerGroup;
+ }
+
+ public String getControllerDLegerPeers() {
+ return controllerDLegerPeers;
+ }
+
+ public void setControllerDLegerPeers(String controllerDLegerPeers) {
+ this.controllerDLegerPeers = controllerDLegerPeers;
+ }
+
+ public String getControllerDLegerSelfId() {
+ return controllerDLegerSelfId;
+ }
+
+ public void setControllerDLegerSelfId(String controllerDLegerSelfId) {
+ this.controllerDLegerSelfId = controllerDLegerSelfId;
+ }
+
+ public String getControllerStorePath() {
+ return controllerStorePath;
+ }
+
+ public void setControllerStorePath(String controllerStorePath) {
+ this.controllerStorePath = controllerStorePath;
+ }
+
+ public boolean isEnableElectUncleanMaster() {
+ return enableElectUncleanMaster;
+ }
+
+ public void setEnableElectUncleanMaster(boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ }
Review Comment:
Do we really need `setter` method? that will make code, which can get object reference, could change the state.
#### Describe in Chinese
该PR中的实体对象有必要定义 setter 方法吗？**setter方法意味着所有获取对象引用的代码都可以修改该对象状态**。而且似乎没有地方用到这些方法。
是否直接在构造函数中为 final 修饰的属性设置值更好、或者使用builder模式。该建议适用所有实体定义。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin merged pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
RongtongJin merged PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857233911
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
Review Comment:
非常赞的写法
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,345 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiPredicate;
+import java.util.function.Predicate;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
Review Comment:
最好传入ControllerConfig，以便于运行时修改参数
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
+
+ this.dLedgerConfig = new DLedgerConfig();
+ this.dLedgerConfig.setGroup(config.getControllerDLegerGroup());
+ this.dLedgerConfig.setPeers(config.getControllerDLegerPeers());
+ this.dLedgerConfig.setSelfId(config.getControllerDLegerSelfId());
+ this.dLedgerConfig.setStoreBaseDir(config.getControllerStorePath());
+ this.dLedgerConfig.setMappedFileSizeForEntryData(config.getMappedFileSize());
+
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(config.isEnableElectUncleanMaster());
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> alterSyncStateSet(AlterSyncStateSetRequestHeader request) {
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> electMaster(final ElectMasterRequestHeader request) {
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> registerBroker(RegisterBrokerRequestHeader request) {
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public RemotingCommand getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr()));
+ }
+
+ /**
+ * Event handler that handle event
+ */
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<RemotingCommand> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+ }
+
+ public <T> CompletableFuture<RemotingCommand> appendEvent(final String name,
+ final Supplier<ControllerResult<T>> supplier, boolean isWriteEvent) {
+ if (isStopped() || !DledgerController.this.roleHandler.isLeaderState()) {
+ final RemotingCommand command = RemotingCommand.createResponseCommand(ResponseCode.CONTROLLER_NOT_LEADER, "The controller is not in leader state");
+ final CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
+ future.complete(command);
+ return future;
+ }
+
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<RemotingCommand> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ if (DledgerController.this.controllerConfig.isProcessReadEvent()) {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ }
+ if (appendSuccess) {
+ final RemotingCommand response = RemotingCommand.createResponseCommandWithHeader(result.getResponseCode(), (CommandCustomHeader) result.getResponse());
+ this.future.complete(response);
+ } else {
+ log.error("Failed to append event to dledger, the response is {}, try cancel the future", result.getResponse());
+ this.future.cancel(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> future() {
+ return this.future;
+ }
+
+ @Override
+ public void handleException(final Throwable t) {
+ log.error("Error happen when handle event {}", this.name, t);
+ this.future.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Role change handler, trigger the startScheduling() and stopScheduling() when role change.
+ */
+ class RoleChangeHandler implements DLedgerLeaderElector.RoleChangeHandler {
+
+ private volatile MemberState.Role currentRole = MemberState.Role.FOLLOWER;
+ private final String selfId;
+ private final ExecutorService executorService = Executors.newSingleThreadExecutor(new ThreadFactoryImpl("DLedgerControllerRoleChangeHandler_"));
+
+ public RoleChangeHandler(final String selfId) {
+ this.selfId = selfId;
+ }
+
+ @Override
+ public void handle(long term, MemberState.Role role) {
+ Runnable runnable = () -> {
+ switch (role) {
+ case CANDIDATE:
+ this.currentRole = MemberState.Role.CANDIDATE;
+ log.info("Controller {} change role to candidate", this.selfId);
+ DledgerController.this.stopScheduling();
+ break;
+ case FOLLOWER:
+ this.currentRole = MemberState.Role.FOLLOWER;
+ log.info("Controller {} change role to Follower, leaderId:{}", this.selfId, getMemberState().getLeaderId());
+ DledgerController.this.stopScheduling();
+ break;
+ case LEADER: {
+ this.currentRole = MemberState.Role.LEADER;
+ log.info("Controller {} change role to leader, try process a initial proposal", this.selfId);
+ // Because the role becomes to leader, but the memory statemachine of the controller is still in the old point,
+ // some committed logs have not been applied. Therefore, we must first process an empty request to dledger,
+ // and after the request is committed, the controller can provide services(startScheduling).
+ int tryTimes = 0;
+ while (true) {
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ try {
+ if (appendToDledgerAndWait(request)) {
+ DledgerController.this.startScheduling();
+ break;
+ }
+ } catch (final Throwable e) {
+ log.error("Error happen when controller leader append initial request to dledger", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ log.error("Controller leader append initial log failed too many times");
+ break;
+ }
Review Comment:
所以连续超过3次会跳出循环吗？是不是应该循环直到append成功
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
+
+ this.dLedgerConfig = new DLedgerConfig();
+ this.dLedgerConfig.setGroup(config.getControllerDLegerGroup());
+ this.dLedgerConfig.setPeers(config.getControllerDLegerPeers());
+ this.dLedgerConfig.setSelfId(config.getControllerDLegerSelfId());
+ this.dLedgerConfig.setStoreBaseDir(config.getControllerStorePath());
+ this.dLedgerConfig.setMappedFileSizeForEntryData(config.getMappedFileSize());
+
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(config.isEnableElectUncleanMaster());
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> alterSyncStateSet(AlterSyncStateSetRequestHeader request) {
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> electMaster(final ElectMasterRequestHeader request) {
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> registerBroker(RegisterBrokerRequestHeader request) {
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public RemotingCommand getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr()));
+ }
+
+ /**
+ * Event handler that handle event
+ */
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<RemotingCommand> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+ }
+
+ public <T> CompletableFuture<RemotingCommand> appendEvent(final String name,
+ final Supplier<ControllerResult<T>> supplier, boolean isWriteEvent) {
+ if (isStopped() || !DledgerController.this.roleHandler.isLeaderState()) {
+ final RemotingCommand command = RemotingCommand.createResponseCommand(ResponseCode.CONTROLLER_NOT_LEADER, "The controller is not in leader state");
+ final CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
+ future.complete(command);
+ return future;
+ }
+
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<RemotingCommand> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ if (DledgerController.this.controllerConfig.isProcessReadEvent()) {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ }
+ if (appendSuccess) {
+ final RemotingCommand response = RemotingCommand.createResponseCommandWithHeader(result.getResponseCode(), (CommandCustomHeader) result.getResponse());
+ this.future.complete(response);
+ } else {
+ log.error("Failed to append event to dledger, the response is {}, try cancel the future", result.getResponse());
+ this.future.cancel(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> future() {
+ return this.future;
+ }
+
+ @Override
+ public void handleException(final Throwable t) {
+ log.error("Error happen when handle event {}", this.name, t);
+ this.future.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Role change handler, trigger the startScheduling() and stopScheduling() when role change.
+ */
+ class RoleChangeHandler implements DLedgerLeaderElector.RoleChangeHandler {
+
+ private volatile MemberState.Role currentRole = MemberState.Role.FOLLOWER;
+ private final String selfId;
+ private final ExecutorService executorService = Executors.newSingleThreadExecutor(new ThreadFactoryImpl("DLedgerControllerRoleChangeHandler_"));
+
+ public RoleChangeHandler(final String selfId) {
+ this.selfId = selfId;
+ }
+
+ @Override
+ public void handle(long term, MemberState.Role role) {
+ Runnable runnable = () -> {
+ switch (role) {
+ case CANDIDATE:
+ this.currentRole = MemberState.Role.CANDIDATE;
+ log.info("Controller {} change role to candidate", this.selfId);
+ DledgerController.this.stopScheduling();
+ break;
+ case FOLLOWER:
+ this.currentRole = MemberState.Role.FOLLOWER;
+ log.info("Controller {} change role to Follower, leaderId:{}", this.selfId, getMemberState().getLeaderId());
+ DledgerController.this.stopScheduling();
+ break;
+ case LEADER: {
+ this.currentRole = MemberState.Role.LEADER;
Review Comment:
如果设置了为Leader，是不是可以马上对event处理了？或者我们应该设置一个标志位，在ready前不对外提供服务
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857059424
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/NamesrvController.java:
##########
@@ -82,24 +85,36 @@ public class NamesrvController {
private ExecutorService defaultExecutor;
private ExecutorService clientRequestExecutor;
+ private ExecutorService controllerRequestExecutor;
private BlockingQueue<Runnable> defaultThreadPoolQueue;
private BlockingQueue<Runnable> clientRequestThreadPoolQueue;
+ private BlockingQueue<Runnable> controllerRequestThreadPoolQueue;
private Configuration configuration;
private FileWatchService fileWatchService;
+ private Controller controller;
+
public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig) {
this(namesrvConfig, nettyServerConfig, new NettyClientConfig());
}
- public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig, NettyClientConfig nettyClientConfig) {
+ public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig,
+ NettyClientConfig nettyClientConfig) {
this.namesrvConfig = namesrvConfig;
this.nettyServerConfig = nettyServerConfig;
this.nettyClientConfig = nettyClientConfig;
this.kvConfigManager = new KVConfigManager(this);
this.brokerHousekeepingService = new BrokerHousekeepingService(this);
- this.routeInfoManager = new RouteInfoManager(namesrvConfig, this);
+ if (namesrvConfig.isStartupController()) {
+ final DLedgerConfig config = new DLedgerConfig();
+ config.setGroup(namesrvConfig.getControllerDLegerGroup());
+ config.setPeers(namesrvConfig.getControllerDLegerPeers());
+ config.setSelfId(namesrvConfig.getControllerDLegerSelfId());
+ this.controller = new DledgerController(config, namesrvConfig.isEnableElectUncleanMaster());
Review Comment:
解决方案: 新增一个 DledgerControllerConfig
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls commented on pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#issuecomment-1113912375
[](https://coveralls.io/builds/48722209)
Coverage increased (+0.2%) to 47.365% when pulling **bbf5c1af6515a7630741e091946e8bc02f841eaa on hzh0425:feature/ha-controller** into **bcce3d32ce7ed53fd1f2851fe14c16f6e68c0082 on apache:5.0.0-beta-dledger-controller**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857260105
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
+
+ this.dLedgerConfig = new DLedgerConfig();
+ this.dLedgerConfig.setGroup(config.getControllerDLegerGroup());
+ this.dLedgerConfig.setPeers(config.getControllerDLegerPeers());
+ this.dLedgerConfig.setSelfId(config.getControllerDLegerSelfId());
+ this.dLedgerConfig.setStoreBaseDir(config.getControllerStorePath());
+ this.dLedgerConfig.setMappedFileSizeForEntryData(config.getMappedFileSize());
+
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(config.isEnableElectUncleanMaster());
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> alterSyncStateSet(AlterSyncStateSetRequestHeader request) {
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> electMaster(final ElectMasterRequestHeader request) {
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> registerBroker(RegisterBrokerRequestHeader request) {
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public RemotingCommand getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr()));
+ }
+
+ /**
+ * Event handler that handle event
+ */
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<RemotingCommand> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+ }
+
+ public <T> CompletableFuture<RemotingCommand> appendEvent(final String name,
+ final Supplier<ControllerResult<T>> supplier, boolean isWriteEvent) {
+ if (isStopped() || !DledgerController.this.roleHandler.isLeaderState()) {
+ final RemotingCommand command = RemotingCommand.createResponseCommand(ResponseCode.CONTROLLER_NOT_LEADER, "The controller is not in leader state");
+ final CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
+ future.complete(command);
+ return future;
+ }
+
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<RemotingCommand> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ if (DledgerController.this.controllerConfig.isProcessReadEvent()) {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ }
+ if (appendSuccess) {
+ final RemotingCommand response = RemotingCommand.createResponseCommandWithHeader(result.getResponseCode(), (CommandCustomHeader) result.getResponse());
+ this.future.complete(response);
+ } else {
+ log.error("Failed to append event to dledger, the response is {}, try cancel the future", result.getResponse());
+ this.future.cancel(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> future() {
+ return this.future;
+ }
+
+ @Override
+ public void handleException(final Throwable t) {
+ log.error("Error happen when handle event {}", this.name, t);
+ this.future.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Role change handler, trigger the startScheduling() and stopScheduling() when role change.
+ */
+ class RoleChangeHandler implements DLedgerLeaderElector.RoleChangeHandler {
+
+ private volatile MemberState.Role currentRole = MemberState.Role.FOLLOWER;
+ private final String selfId;
+ private final ExecutorService executorService = Executors.newSingleThreadExecutor(new ThreadFactoryImpl("DLedgerControllerRoleChangeHandler_"));
+
+ public RoleChangeHandler(final String selfId) {
+ this.selfId = selfId;
+ }
+
+ @Override
+ public void handle(long term, MemberState.Role role) {
+ Runnable runnable = () -> {
+ switch (role) {
+ case CANDIDATE:
+ this.currentRole = MemberState.Role.CANDIDATE;
+ log.info("Controller {} change role to candidate", this.selfId);
+ DledgerController.this.stopScheduling();
+ break;
+ case FOLLOWER:
+ this.currentRole = MemberState.Role.FOLLOWER;
+ log.info("Controller {} change role to Follower, leaderId:{}", this.selfId, getMemberState().getLeaderId());
+ DledgerController.this.stopScheduling();
+ break;
+ case LEADER: {
+ this.currentRole = MemberState.Role.LEADER;
+ log.info("Controller {} change role to leader, try process a initial proposal", this.selfId);
+ // Because the role becomes to leader, but the memory statemachine of the controller is still in the old point,
+ // some committed logs have not been applied. Therefore, we must first process an empty request to dledger,
+ // and after the request is committed, the controller can provide services(startScheduling).
+ int tryTimes = 0;
+ while (true) {
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ try {
+ if (appendToDledgerAndWait(request)) {
+ DledgerController.this.startScheduling();
+ break;
+ }
+ } catch (final Throwable e) {
+ log.error("Error happen when controller leader append initial request to dledger", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ log.error("Controller leader append initial log failed too many times");
+ break;
+ }
Review Comment:
可能刚转换成 leader 的时候, 需要一段时间才能 append 成功. 但是并不知道成功的边界. 会不会有可能一直不成功?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857059501
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
讨论方案: 在 role change 到 leader 时, 主动发起一次空的提案. 等该提案提交之后, 才代表controller 的元数据都已经恢复了.然后才能对外提供服务.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857059556
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/InSyncReplicasInfo.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * In sync replicas info, manages the master and syncStateSet of a broker.
+ */
+public class InSyncReplicasInfo {
+ private final String clusterName;
+ private final String brokerName;
+
+ private Set<String/*Address*/> syncStateSet;
+ private int syncStateSetEpoch;
+
+ private String masterAddress;
+ private int masterEpoch;
+ // Because when a Broker becomes a master, its id needs to be assigned a value of 0.
+ // We need to record it's originId so that when it becomes a follower again, we can find its original id.
+ private long masterOriginId;
Review Comment:
讨论方案: 删除 masterOriginId. 不改变 brokerIdTable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dugenkui03 commented on a diff in pull request #4195: Dledger controller
Posted by GitBox <gi...@apache.org>.
dugenkui03 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r855767880
##########
common/src/main/java/org/apache/rocketmq/common/namesrv/NamesrvConfig.java:
##########
@@ -200,4 +221,68 @@ public boolean isEnableAllTopicList() {
public void setEnableAllTopicList(boolean enableAllTopicList) {
this.enableAllTopicList = enableAllTopicList;
}
+
+ public boolean isStartupController() {
+ return isStartupController;
+ }
+
+ public void setStartupController(boolean startupController) {
+ isStartupController = startupController;
+ }
+
+ public int getControllerThreadPoolNums() {
+ return controllerThreadPoolNums;
+ }
+
+ public void setControllerThreadPoolNums(int controllerThreadPoolNums) {
+ this.controllerThreadPoolNums = controllerThreadPoolNums;
+ }
+
+ public int getControllerRequestThreadPoolQueueCapacity() {
+ return controllerRequestThreadPoolQueueCapacity;
+ }
+
+ public void setControllerRequestThreadPoolQueueCapacity(int controllerRequestThreadPoolQueueCapacity) {
+ this.controllerRequestThreadPoolQueueCapacity = controllerRequestThreadPoolQueueCapacity;
+ }
+
+ public String getControllerDLegerGroup() {
+ return controllerDLegerGroup;
+ }
+
+ public void setControllerDLegerGroup(String controllerDLegerGroup) {
+ this.controllerDLegerGroup = controllerDLegerGroup;
+ }
+
+ public String getControllerDLegerPeers() {
+ return controllerDLegerPeers;
+ }
+
+ public void setControllerDLegerPeers(String controllerDLegerPeers) {
+ this.controllerDLegerPeers = controllerDLegerPeers;
+ }
+
+ public String getControllerDLegerSelfId() {
+ return controllerDLegerSelfId;
+ }
+
+ public void setControllerDLegerSelfId(String controllerDLegerSelfId) {
+ this.controllerDLegerSelfId = controllerDLegerSelfId;
+ }
+
+ public String getControllerStorePath() {
+ return controllerStorePath;
+ }
+
+ public void setControllerStorePath(String controllerStorePath) {
+ this.controllerStorePath = controllerStorePath;
+ }
+
+ public boolean isEnableElectUncleanMaster() {
+ return enableElectUncleanMaster;
+ }
+
+ public void setEnableElectUncleanMaster(boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ }
Review Comment:
Do we really need `setter` method? that will make code, which can get object reference, could change the state.
#### Describe in Chinese
该PR中的实体对象有必要定义 setter 方法吗？**setter方法意味着所有获取对象引用的代码都可以修改该对象状态**。而且似乎没有地方用到这些方法。
是否直接在构造函数中为 final 修饰的属性设置值更好、或者使用builder模式。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dugenkui03 commented on a diff in pull request #4195: Dledger controller
Posted by GitBox <gi...@apache.org>.
dugenkui03 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r855769402
##########
common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/controller/AlterSyncStateSetRequestHeader.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.protocol.header.namesrv.controller;
+
+import java.util.Set;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.exception.RemotingCommandException;
+
+public class AlterSyncStateSetRequestHeader implements CommandCustomHeader {
+ private String brokerName;
+ private String masterAddress;
+ private int masterEpoch;
+ private Set<String> newSyncStateSet;
+ private int syncStateSetEpoch;
+
+ public AlterSyncStateSetRequestHeader(String brokerName, String masterAddress, int masterEpoch,
+ Set<String> newSyncStateSet, int syncStateSetEpoch) {
+ this.brokerName = brokerName;
+ this.masterAddress = masterAddress;
+ this.masterEpoch = masterEpoch;
+ this.newSyncStateSet = newSyncStateSet;
+ this.syncStateSetEpoch = syncStateSetEpoch;
Review Comment:
Is copy defensive better?
#### Describe in Chinese
这里使用保护性拷贝更安全。getNewSyncStateSet同理。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857055450
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
+ } else {
+ brokerId = brokerIdTable.get(brokerAddress);
+ }
+ response.setBrokerId(brokerId);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch());
+
+ if (replicasInfo.isMasterAlive()) {
+ // If the master is alive, just return master info.
+ response.setMasterAddress(replicasInfo.getMasterAddress());
+ return result;
+ } else {
+ // If the master is not alive, we should elect a new master:
+ // Case1: This replicas was in sync state set list
+ // Case2: The option {EnableElectUncleanMaster} is true
+ canBeElectedAsMaster = replicasInfo.getSyncStateSet().contains(brokerAddress) || this.enableElectUncleanMaster;
+ }
+ } else {
+ // If the broker's metadata does not exist in the state machine, the replicas can be elected as master directly.
+ canBeElectedAsMaster = true;
+ }
+
+ if (canBeElectedAsMaster) {
+ int masterEpoch = this.inSyncReplicasInfoTable.containsKey(brokerName) ?
+ this.inSyncReplicasInfoTable.get(brokerName).getMasterEpoch() + 1 : 1;
+ response.setMasterAddress(request.getBrokerAddress());
+ response.setMasterEpoch(masterEpoch);
+ response.setBrokerId(0);
+
+ final ElectMasterEvent event = new ElectMasterEvent(true, brokerName, brokerAddress, request.getClusterName());
+ result.addEvent(event);
+ return result;
+ }
Review Comment:
在 handleElectMaster 里, 当 brokerName 不存在, 会自动会让该 replicas 的 brokerId = 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857054342
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
这需要依靠 dledger snapshot 的支持, 从最近一次 snapshot 恢复状态, 然后恢复之后的日志.
但是目前 dledger 不支持该能力, 所以只能从头开始回放日志.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857282933
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
+
+ this.dLedgerConfig = new DLedgerConfig();
+ this.dLedgerConfig.setGroup(config.getControllerDLegerGroup());
+ this.dLedgerConfig.setPeers(config.getControllerDLegerPeers());
+ this.dLedgerConfig.setSelfId(config.getControllerDLegerSelfId());
+ this.dLedgerConfig.setStoreBaseDir(config.getControllerStorePath());
+ this.dLedgerConfig.setMappedFileSizeForEntryData(config.getMappedFileSize());
+
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(config.isEnableElectUncleanMaster());
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> alterSyncStateSet(AlterSyncStateSetRequestHeader request) {
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> electMaster(final ElectMasterRequestHeader request) {
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> registerBroker(RegisterBrokerRequestHeader request) {
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public RemotingCommand getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr()));
+ }
+
+ /**
+ * Event handler that handle event
+ */
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<RemotingCommand> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+ }
+
+ public <T> CompletableFuture<RemotingCommand> appendEvent(final String name,
+ final Supplier<ControllerResult<T>> supplier, boolean isWriteEvent) {
+ if (isStopped() || !DledgerController.this.roleHandler.isLeaderState()) {
+ final RemotingCommand command = RemotingCommand.createResponseCommand(ResponseCode.CONTROLLER_NOT_LEADER, "The controller is not in leader state");
+ final CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
+ future.complete(command);
+ return future;
+ }
+
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<RemotingCommand> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ if (DledgerController.this.controllerConfig.isProcessReadEvent()) {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ }
+ if (appendSuccess) {
+ final RemotingCommand response = RemotingCommand.createResponseCommandWithHeader(result.getResponseCode(), (CommandCustomHeader) result.getResponse());
+ this.future.complete(response);
+ } else {
+ log.error("Failed to append event to dledger, the response is {}, try cancel the future", result.getResponse());
+ this.future.cancel(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> future() {
+ return this.future;
+ }
+
+ @Override
+ public void handleException(final Throwable t) {
+ log.error("Error happen when handle event {}", this.name, t);
+ this.future.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Role change handler, trigger the startScheduling() and stopScheduling() when role change.
+ */
+ class RoleChangeHandler implements DLedgerLeaderElector.RoleChangeHandler {
+
+ private volatile MemberState.Role currentRole = MemberState.Role.FOLLOWER;
+ private final String selfId;
+ private final ExecutorService executorService = Executors.newSingleThreadExecutor(new ThreadFactoryImpl("DLedgerControllerRoleChangeHandler_"));
+
+ public RoleChangeHandler(final String selfId) {
+ this.selfId = selfId;
+ }
+
+ @Override
+ public void handle(long term, MemberState.Role role) {
+ Runnable runnable = () -> {
+ switch (role) {
+ case CANDIDATE:
+ this.currentRole = MemberState.Role.CANDIDATE;
+ log.info("Controller {} change role to candidate", this.selfId);
+ DledgerController.this.stopScheduling();
+ break;
+ case FOLLOWER:
+ this.currentRole = MemberState.Role.FOLLOWER;
+ log.info("Controller {} change role to Follower, leaderId:{}", this.selfId, getMemberState().getLeaderId());
+ DledgerController.this.stopScheduling();
+ break;
+ case LEADER: {
+ this.currentRole = MemberState.Role.LEADER;
+ log.info("Controller {} change role to leader, try process a initial proposal", this.selfId);
+ // Because the role becomes to leader, but the memory statemachine of the controller is still in the old point,
+ // some committed logs have not been applied. Therefore, we must first process an empty request to dledger,
+ // and after the request is committed, the controller can provide services(startScheduling).
+ int tryTimes = 0;
+ while (true) {
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ try {
+ if (appendToDledgerAndWait(request)) {
+ DledgerController.this.startScheduling();
+ break;
+ }
+ } catch (final Throwable e) {
+ log.error("Error happen when controller leader append initial request to dledger", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ log.error("Controller leader append initial log failed too many times");
+ break;
+ }
Review Comment:
是的，但我认为成功前不应该提供服务，或者每失败超过x次打一次日志来提醒用户。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] coveralls commented on pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
coveralls commented on PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#issuecomment-1113912377
[](https://coveralls.io/builds/48722209)
Coverage increased (+0.2%) to 47.365% when pulling **bbf5c1af6515a7630741e091946e8bc02f841eaa on hzh0425:feature/ha-controller** into **bcce3d32ce7ed53fd1f2851fe14c16f6e68c0082 on apache:5.0.0-beta-dledger-controller**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on pull request #4195: Feature/ha controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#issuecomment-1105974421
@RongtongJin Hi,task a look please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] codecov-commenter commented on pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#issuecomment-1113912344
# [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4195?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#4195](https://codecov.io/gh/apache/rocketmq/pull/4195?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bbf5c1a) into [5.0.0-beta-dledger-controller](https://codecov.io/gh/apache/rocketmq/commit/bcce3d32ce7ed53fd1f2851fe14c16f6e68c0082?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bcce3d3) will **increase** coverage by `0.06%`.
> The diff coverage is `53.40%`.
```diff
@@ Coverage Diff @@
## 5.0.0-beta-dledger-controller #4195 +/- ##
===================================================================
+ Coverage 43.13% 43.19% +0.06%
- Complexity 6025 6121 +96
===================================================================
Files 795 818 +23
Lines 56859 57559 +700
Branches 7787 7852 +65
===================================================================
+ Hits 24524 24864 +340
- Misses 29141 29452 +311
- Partials 3194 3243 +49
```
| [Impacted Files](https://codecov.io/gh/apache/rocketmq/pull/4195?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/rocketmq/common/constant/LoggerName.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vY29uc3RhbnQvTG9nZ2VyTmFtZS5qYXZh) | `0.00% <ø> (ø)` | |
| [...ache/rocketmq/common/namesrv/ControllerConfig.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vbmFtZXNydi9Db250cm9sbGVyQ29uZmlnLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...g/apache/rocketmq/common/protocol/RequestCode.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvUmVxdWVzdENvZGUuamF2YQ==) | `0.00% <ø> (ø)` | |
| [.../apache/rocketmq/common/protocol/ResponseCode.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvUmVzcG9uc2VDb2RlLmphdmE=) | `0.00% <ø> (ø)` | |
| [...srv/controller/AlterSyncStateSetRequestHeader.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvaGVhZGVyL25hbWVzcnYvY29udHJvbGxlci9BbHRlclN5bmNTdGF0ZVNldFJlcXVlc3RIZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...rv/controller/AlterSyncStateSetResponseHeader.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvaGVhZGVyL25hbWVzcnYvY29udHJvbGxlci9BbHRlclN5bmNTdGF0ZVNldFJlc3BvbnNlSGVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...r/namesrv/controller/ElectMasterRequestHeader.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvaGVhZGVyL25hbWVzcnYvY29udHJvbGxlci9FbGVjdE1hc3RlclJlcXVlc3RIZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../namesrv/controller/ElectMasterResponseHeader.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvaGVhZGVyL25hbWVzcnYvY29udHJvbGxlci9FbGVjdE1hc3RlclJlc3BvbnNlSGVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../namesrv/controller/GetMetaDataResponseHeader.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvaGVhZGVyL25hbWVzcnYvY29udHJvbGxlci9HZXRNZXRhRGF0YVJlc3BvbnNlSGVhZGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...amesrv/controller/GetReplicaInfoRequestHeader.java](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9yb2NrZXRtcS9jb21tb24vcHJvdG9jb2wvaGVhZGVyL25hbWVzcnYvY29udHJvbGxlci9HZXRSZXBsaWNhSW5mb1JlcXVlc3RIZWFkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| ... and [33 more](https://codecov.io/gh/apache/rocketmq/pull/4195/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/rocketmq/pull/4195?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/rocketmq/pull/4195?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bcce3d3...bbf5c1a](https://codecov.io/gh/apache/rocketmq/pull/4195?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r856893803
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
好的，那我们就得去想一下如何recover，而在recover完成前是无法提供服务的，因为内存中的数据是不可靠的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857260249
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
+
+ this.dLedgerConfig = new DLedgerConfig();
+ this.dLedgerConfig.setGroup(config.getControllerDLegerGroup());
+ this.dLedgerConfig.setPeers(config.getControllerDLegerPeers());
+ this.dLedgerConfig.setSelfId(config.getControllerDLegerSelfId());
+ this.dLedgerConfig.setStoreBaseDir(config.getControllerStorePath());
+ this.dLedgerConfig.setMappedFileSizeForEntryData(config.getMappedFileSize());
+
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(config.isEnableElectUncleanMaster());
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> alterSyncStateSet(AlterSyncStateSetRequestHeader request) {
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> electMaster(final ElectMasterRequestHeader request) {
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> registerBroker(RegisterBrokerRequestHeader request) {
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public RemotingCommand getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr()));
+ }
+
+ /**
+ * Event handler that handle event
+ */
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<RemotingCommand> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+ }
+
+ public <T> CompletableFuture<RemotingCommand> appendEvent(final String name,
+ final Supplier<ControllerResult<T>> supplier, boolean isWriteEvent) {
+ if (isStopped() || !DledgerController.this.roleHandler.isLeaderState()) {
+ final RemotingCommand command = RemotingCommand.createResponseCommand(ResponseCode.CONTROLLER_NOT_LEADER, "The controller is not in leader state");
+ final CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
+ future.complete(command);
+ return future;
+ }
+
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<RemotingCommand> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ if (DledgerController.this.controllerConfig.isProcessReadEvent()) {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ }
+ if (appendSuccess) {
+ final RemotingCommand response = RemotingCommand.createResponseCommandWithHeader(result.getResponseCode(), (CommandCustomHeader) result.getResponse());
+ this.future.complete(response);
+ } else {
+ log.error("Failed to append event to dledger, the response is {}, try cancel the future", result.getResponse());
+ this.future.cancel(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> future() {
+ return this.future;
+ }
+
+ @Override
+ public void handleException(final Throwable t) {
+ log.error("Error happen when handle event {}", this.name, t);
+ this.future.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Role change handler, trigger the startScheduling() and stopScheduling() when role change.
+ */
+ class RoleChangeHandler implements DLedgerLeaderElector.RoleChangeHandler {
+
+ private volatile MemberState.Role currentRole = MemberState.Role.FOLLOWER;
+ private final String selfId;
+ private final ExecutorService executorService = Executors.newSingleThreadExecutor(new ThreadFactoryImpl("DLedgerControllerRoleChangeHandler_"));
+
+ public RoleChangeHandler(final String selfId) {
+ this.selfId = selfId;
+ }
+
+ @Override
+ public void handle(long term, MemberState.Role role) {
+ Runnable runnable = () -> {
+ switch (role) {
+ case CANDIDATE:
+ this.currentRole = MemberState.Role.CANDIDATE;
+ log.info("Controller {} change role to candidate", this.selfId);
+ DledgerController.this.stopScheduling();
+ break;
+ case FOLLOWER:
+ this.currentRole = MemberState.Role.FOLLOWER;
+ log.info("Controller {} change role to Follower, leaderId:{}", this.selfId, getMemberState().getLeaderId());
+ DledgerController.this.stopScheduling();
+ break;
+ case LEADER: {
+ this.currentRole = MemberState.Role.LEADER;
Review Comment:
嗯嗯, 这里有问题, 应该在下面 append 成功后, 才标志为 leader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857283382
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiPredicate;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.ServiceThread;
+import org.apache.rocketmq.common.ThreadFactoryImpl;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.namesrv.ControllerConfig;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.NamesrvController;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.CommandCustomHeader;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final ControllerConfig controllerConfig;
+ private final DLedgerConfig dLedgerConfig;
+ private final NamesrvController namesrvController;
+ // Usr for checking whether the broker is alive
+ private final BiPredicate<String, String> brokerAlivePredicate;
+
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final ControllerConfig config, final NamesrvController namesrvController) {
+ this.controllerConfig = config;
+ this.eventSerializer = new EventSerializer();
+ this.scheduler = new EventScheduler();
+ this.namesrvController = namesrvController;
+ if (namesrvController == null) {
+ this.brokerAlivePredicate = (cluster, address) -> true;
+ } else {
+ this.brokerAlivePredicate = (cluster, address) -> namesrvController.getRouteInfoManager().isBrokerAlive(cluster, address);
+ }
+
+ this.dLedgerConfig = new DLedgerConfig();
+ this.dLedgerConfig.setGroup(config.getControllerDLegerGroup());
+ this.dLedgerConfig.setPeers(config.getControllerDLegerPeers());
+ this.dLedgerConfig.setSelfId(config.getControllerDLegerSelfId());
+ this.dLedgerConfig.setStoreBaseDir(config.getControllerStorePath());
+ this.dLedgerConfig.setMappedFileSizeForEntryData(config.getMappedFileSize());
+
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(config.isEnableElectUncleanMaster());
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> alterSyncStateSet(AlterSyncStateSetRequestHeader request) {
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> electMaster(final ElectMasterRequestHeader request) {
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request, this.brokerAlivePredicate), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> registerBroker(RegisterBrokerRequestHeader request) {
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public RemotingCommand getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr()));
+ }
+
+ /**
+ * Event handler that handle event
+ */
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<RemotingCommand> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+ }
+
+ public <T> CompletableFuture<RemotingCommand> appendEvent(final String name,
+ final Supplier<ControllerResult<T>> supplier, boolean isWriteEvent) {
+ if (isStopped() || !DledgerController.this.roleHandler.isLeaderState()) {
+ final RemotingCommand command = RemotingCommand.createResponseCommand(ResponseCode.CONTROLLER_NOT_LEADER, "The controller is not in leader state");
+ final CompletableFuture<RemotingCommand> future = new CompletableFuture<>();
+ future.complete(command);
+ return future;
+ }
+
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<RemotingCommand> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ if (DledgerController.this.controllerConfig.isProcessReadEvent()) {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ }
+ if (appendSuccess) {
+ final RemotingCommand response = RemotingCommand.createResponseCommandWithHeader(result.getResponseCode(), (CommandCustomHeader) result.getResponse());
+ this.future.complete(response);
+ } else {
+ log.error("Failed to append event to dledger, the response is {}, try cancel the future", result.getResponse());
+ this.future.cancel(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<RemotingCommand> future() {
+ return this.future;
+ }
+
+ @Override
+ public void handleException(final Throwable t) {
+ log.error("Error happen when handle event {}", this.name, t);
+ this.future.completeExceptionally(t);
+ }
+ }
+
+ /**
+ * Role change handler, trigger the startScheduling() and stopScheduling() when role change.
+ */
+ class RoleChangeHandler implements DLedgerLeaderElector.RoleChangeHandler {
+
+ private volatile MemberState.Role currentRole = MemberState.Role.FOLLOWER;
+ private final String selfId;
+ private final ExecutorService executorService = Executors.newSingleThreadExecutor(new ThreadFactoryImpl("DLedgerControllerRoleChangeHandler_"));
+
+ public RoleChangeHandler(final String selfId) {
+ this.selfId = selfId;
+ }
+
+ @Override
+ public void handle(long term, MemberState.Role role) {
+ Runnable runnable = () -> {
+ switch (role) {
+ case CANDIDATE:
+ this.currentRole = MemberState.Role.CANDIDATE;
+ log.info("Controller {} change role to candidate", this.selfId);
+ DledgerController.this.stopScheduling();
+ break;
+ case FOLLOWER:
+ this.currentRole = MemberState.Role.FOLLOWER;
+ log.info("Controller {} change role to Follower, leaderId:{}", this.selfId, getMemberState().getLeaderId());
+ DledgerController.this.stopScheduling();
+ break;
+ case LEADER: {
+ this.currentRole = MemberState.Role.LEADER;
+ log.info("Controller {} change role to leader, try process a initial proposal", this.selfId);
+ // Because the role becomes to leader, but the memory statemachine of the controller is still in the old point,
+ // some committed logs have not been applied. Therefore, we must first process an empty request to dledger,
+ // and after the request is committed, the controller can provide services(startScheduling).
+ int tryTimes = 0;
+ while (true) {
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ try {
+ if (appendToDledgerAndWait(request)) {
+ DledgerController.this.startScheduling();
+ break;
+ }
+ } catch (final Throwable e) {
+ log.error("Error happen when controller leader append initial request to dledger", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ log.error("Controller leader append initial log failed too many times");
+ break;
+ }
Review Comment:
good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] dugenkui03 commented on pull request #4195: Dledger controller
Posted by GitBox <gi...@apache.org>.
dugenkui03 commented on PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#issuecomment-1106017550
Some suggestion, details in review:
1. remove unnessary setter;
2. return copy or unmodified view for getter;
3. use defensive copy in constructor;
## Describe in Chinese
评论中提交了三个建议、适用所有实体定义。请评估。
1. 移除不必要的 setter、用 final+constroctor 或者 final + builder 替代；
2. getter中返回保护性拷贝或者 不可修改视图；
3. 构造函数对于列表使用保护性拷贝。
这些建议的目的均为使得对象安全的创建或者发布，避免可以获取到对象引用或者对象集合字段引用的不可信代码、不恰当的使用导致对象状态被破坏。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r856890750
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.common.ServiceThread;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final DLedgerConfig dLedgerConfig;
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final DLedgerConfig dLedgerConfig, final boolean isEnableElectUncleanMaster) {
+ this.dLedgerConfig = dLedgerConfig;
+
+ this.eventSerializer = new EventSerializer();
+
+ this.scheduler = new EventScheduler();
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(isEnableElectUncleanMaster);
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
Review Comment:
建议使用org.apache.rocketmq.common.ServiceThread而非org.apache.rocketmq.remoting.common.ServiceThread，不然EventScheduler在shutdown后将不能再start
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/Controller.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+
+/**
+ * The api for controller
+ */
+public interface Controller {
+
+ interface EventHandler<T> {
+ /**
+ * Run the controller event
+ */
+ void run() throws Throwable;
+
+ /**
+ * Return the completableFuture
+ */
+ CompletableFuture<T> future();
+
+ /**
+ * Handle Exception.
+ */
+ void handleException(final Throwable t);
+ }
Review Comment:
或者可以另起一个文件写这个接口
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.common.ServiceThread;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final DLedgerConfig dLedgerConfig;
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final DLedgerConfig dLedgerConfig, final boolean isEnableElectUncleanMaster) {
+ this.dLedgerConfig = dLedgerConfig;
+
+ this.eventSerializer = new EventSerializer();
+
+ this.scheduler = new EventScheduler();
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(isEnableElectUncleanMaster);
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ AlterSyncStateSetRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject alterSyncStateSet request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request), true);
+ }
+
+ @Override
+ public CompletableFuture<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject electMaster request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RegisterBrokerResponseHeader> registerBroker(RegisterBrokerRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject registerBroker request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<GetReplicaInfoResponseHeader> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject getReplicaInfo request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public GetMetaDataResponseHeader getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr());
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+
+ }
+
+ public <T> CompletableFuture<T> appendEvent(final String name, final Supplier<ControllerResult<T>> supplier,
+ boolean isWriteEvent) {
+ if (isStopped()) {
+ return null;
+ }
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<T> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
Review Comment:
getReplicaInfo不建议每次走读日志，建议直接读内存，最终一致性。因为这个操作在broker端会比较频繁，如果broker很多的情况下，对controller的压力很大
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
好的，那我们就得去想一下如何recover，而在recover完成前是无法提供服务的，因为内存中的数据是不可靠的
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/NamesrvController.java:
##########
@@ -238,6 +270,11 @@ private void registerProcessor() {
this.remotingServer.registerProcessor(RequestCode.GET_ROUTEINFO_BY_TOPIC, clientRequestProcessor, this.clientRequestExecutor);
this.remotingServer.registerDefaultProcessor(new DefaultRequestProcessor(this), this.defaultExecutor);
+
+ if (namesrvConfig.isStartupController()) {
+ final ControllerRequestProcessor controllerRequestProcessor = new ControllerRequestProcessor(this.controller);
+ this.remotingServer.registerProcessor(RequestCode.CONTROLLER_REQUEST, controllerRequestProcessor, this.controllerRequestExecutor);
Review Comment:
这里是不是忘了注册其他的requestCode
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/NamesrvController.java:
##########
@@ -82,24 +85,36 @@ public class NamesrvController {
private ExecutorService defaultExecutor;
private ExecutorService clientRequestExecutor;
+ private ExecutorService controllerRequestExecutor;
private BlockingQueue<Runnable> defaultThreadPoolQueue;
private BlockingQueue<Runnable> clientRequestThreadPoolQueue;
+ private BlockingQueue<Runnable> controllerRequestThreadPoolQueue;
private Configuration configuration;
private FileWatchService fileWatchService;
+ private Controller controller;
+
public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig) {
this(namesrvConfig, nettyServerConfig, new NettyClientConfig());
}
- public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig, NettyClientConfig nettyClientConfig) {
+ public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig,
+ NettyClientConfig nettyClientConfig) {
this.namesrvConfig = namesrvConfig;
this.nettyServerConfig = nettyServerConfig;
this.nettyClientConfig = nettyClientConfig;
this.kvConfigManager = new KVConfigManager(this);
this.brokerHousekeepingService = new BrokerHousekeepingService(this);
- this.routeInfoManager = new RouteInfoManager(namesrvConfig, this);
+ if (namesrvConfig.isStartupController()) {
+ final DLedgerConfig config = new DLedgerConfig();
+ config.setGroup(namesrvConfig.getControllerDLegerGroup());
+ config.setPeers(namesrvConfig.getControllerDLegerPeers());
+ config.setSelfId(namesrvConfig.getControllerDLegerSelfId());
+ this.controller = new DledgerController(config, namesrvConfig.isEnableElectUncleanMaster());
Review Comment:
这样的构造方式可能会导致namesrvConfig在运行时通过admin工具变更enableElectUncleanMaster配置，但不起作用。或者应该传入controllerConfig（namesrvConfig）
##########
common/src/main/java/org/apache/rocketmq/common/namesrv/NamesrvConfig.java:
##########
@@ -73,6 +73,27 @@ public class NamesrvConfig {
private volatile boolean enableAllTopicList = false;
+ /**
+ * Dledger controller config
+ */
+ private boolean isStartupController = false;
+ /**
+ * Indicates the nums of thread to handle broker or operation requests, like REGISTER_BROKER.
+ */
+ private int controllerThreadPoolNums = 16;
+ /**
+ * Indicates the capacity of queue to hold client requests.
+ */
+ private int controllerRequestThreadPoolQueueCapacity = 50000;
+ private String controllerDLegerGroup;
+ private String controllerDLegerPeers;
+ private String controllerDLegerSelfId;
+ private String controllerStorePath = System.getProperty("user.home") + File.separator + "DledgerController";
+ /**
+ * Whether the controller can elect a master which is not in the syncStateSet.
+ */
+ private boolean enableElectUncleanMaster = false;
Review Comment:
建议单独搞个Controller的配置，在启动的时候（createNamesrvController）MixAll.properties2Object(properties, controllerConfig);即可
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/InSyncReplicasInfo.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * In sync replicas info, manages the master and syncStateSet of a broker.
+ */
+public class InSyncReplicasInfo {
+ private final String clusterName;
+ private final String brokerName;
+
+ private Set<String/*Address*/> syncStateSet;
+ private int syncStateSetEpoch;
+
+ private String masterAddress;
+ private int masterEpoch;
+ // Because when a Broker becomes a master, its id needs to be assigned a value of 0.
+ // We need to record it's originId so that when it becomes a follower again, we can find its original id.
+ private long masterOriginId;
Review Comment:
或许我们不需要记录masterOriginId，只要保持一个规则，master的id为0，其他的id为原编号，而brokerIdTable也只需要记录分配的编号即可，不需要一直变换
##########
common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/controller/ErrorCodes.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.protocol.header.namesrv.controller;
+
+/**
+ * @author hzh
+ * @email 642256541@qq.com
+ * @date 2022/4/16 20:01
+ */
+public enum ErrorCodes {
+
+ NONE((short) 0, "No error"),
+ FENCED_LEADER_EPOCH((short) 1, "The leader epoch in the request is older than the current epoch"),
+ FENCED_SYNC_STATE_SET_EPOCH((short) 2, "The syncStateSet epoch in the request is older than the current epoch"),
+ INVALID_REQUEST((short) 3, "The request is invalid"),
+ MASTER_NOT_AVAILABLE((short) 4, "There is no available master for this broker.");
Review Comment:
建议和ResponseCode做一些融合
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/InSyncReplicasInfo.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * In sync replicas info, manages the master and syncStateSet of a broker.
+ */
+public class InSyncReplicasInfo {
+ private final String clusterName;
+ private final String brokerName;
+
+ private Set<String/*Address*/> syncStateSet;
+ private int syncStateSetEpoch;
+
+ private String masterAddress;
+ private int masterEpoch;
+ // Because when a Broker becomes a master, its id needs to be assigned a value of 0.
+ // We need to record it's originId so that when it becomes a follower again, we can find its original id.
+ private long masterOriginId;
+
+ public InSyncReplicasInfo(String clusterName, String brokerName, String masterAddress) {
+ this.clusterName = clusterName;
+ this.brokerName = brokerName;
+ this.masterAddress = masterAddress;
+ this.masterEpoch = 1;
+ // The first master is the first online broker
+ this.masterOriginId = 1;
+ this.syncStateSet = new HashSet<>();
+ this.syncStateSet.add(masterAddress);
+ this.syncStateSetEpoch = 1;
+ }
+
+ public void updateMasterInfo(String masterAddress, long masterOriginId) {
+ this.masterAddress = masterAddress;
+ this.masterOriginId = masterOriginId;
+ this.masterEpoch++;
+ }
+
+ public void updateSyncStateSetInfo(Set<String> newSyncStateSet) {
+ this.syncStateSet = newSyncStateSet;
+ this.syncStateSetEpoch++;
+ }
+
+ public boolean isMasterAlive() {
Review Comment:
或者叫isMasterExist更好
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
Review Comment:
利用Mix.MASTER_ID即可
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
+ } else {
+ brokerId = brokerIdTable.get(brokerAddress);
+ }
+ response.setBrokerId(brokerId);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch());
+
+ if (replicasInfo.isMasterAlive()) {
+ // If the master is alive, just return master info.
+ response.setMasterAddress(replicasInfo.getMasterAddress());
+ return result;
+ } else {
+ // If the master is not alive, we should elect a new master:
+ // Case1: This replicas was in sync state set list
+ // Case2: The option {EnableElectUncleanMaster} is true
+ canBeElectedAsMaster = replicasInfo.getSyncStateSet().contains(brokerAddress) || this.enableElectUncleanMaster;
+ }
+ } else {
+ // If the broker's metadata does not exist in the state machine, the replicas can be elected as master directly.
+ canBeElectedAsMaster = true;
+ }
+
+ if (canBeElectedAsMaster) {
+ int masterEpoch = this.inSyncReplicasInfoTable.containsKey(brokerName) ?
+ this.inSyncReplicasInfoTable.get(brokerName).getMasterEpoch() + 1 : 1;
+ response.setMasterAddress(request.getBrokerAddress());
+ response.setMasterEpoch(masterEpoch);
+ response.setBrokerId(0);
+
+ final ElectMasterEvent event = new ElectMasterEvent(true, brokerName, brokerAddress, request.getClusterName());
+ result.addEvent(event);
+ return result;
+ }
Review Comment:
这里似乎brokerName不存在，就不会为对应的broker去申请brokerId
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
+ } else {
+ brokerId = brokerIdTable.get(brokerAddress);
+ }
+ response.setBrokerId(brokerId);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch());
+
+ if (replicasInfo.isMasterAlive()) {
+ // If the master is alive, just return master info.
+ response.setMasterAddress(replicasInfo.getMasterAddress());
+ return result;
+ } else {
+ // If the master is not alive, we should elect a new master:
+ // Case1: This replicas was in sync state set list
+ // Case2: The option {EnableElectUncleanMaster} is true
+ canBeElectedAsMaster = replicasInfo.getSyncStateSet().contains(brokerAddress) || this.enableElectUncleanMaster;
Review Comment:
enableElectUncleanMaster应该是在electMaster时发现SycnStatSet中没有可选broker时，从所有存活的broker中挑选
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
+ } else {
+ brokerId = brokerIdTable.get(brokerAddress);
+ }
+ response.setBrokerId(brokerId);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch());
+
+ if (replicasInfo.isMasterAlive()) {
+ // If the master is alive, just return master info.
+ response.setMasterAddress(replicasInfo.getMasterAddress());
+ return result;
+ } else {
+ // If the master is not alive, we should elect a new master:
+ // Case1: This replicas was in sync state set list
+ // Case2: The option {EnableElectUncleanMaster} is true
+ canBeElectedAsMaster = replicasInfo.getSyncStateSet().contains(brokerAddress) || this.enableElectUncleanMaster;
+ }
+ } else {
+ // If the broker's metadata does not exist in the state machine, the replicas can be elected as master directly.
+ canBeElectedAsMaster = true;
+ }
+
+ if (canBeElectedAsMaster) {
+ int masterEpoch = this.inSyncReplicasInfoTable.containsKey(brokerName) ?
+ this.inSyncReplicasInfoTable.get(brokerName).getMasterEpoch() + 1 : 1;
+ response.setMasterAddress(request.getBrokerAddress());
+ response.setMasterEpoch(masterEpoch);
+ response.setBrokerId(0);
+
+ final ElectMasterEvent event = new ElectMasterEvent(true, brokerName, brokerAddress, request.getClusterName());
+ result.addEvent(event);
+ return result;
+ }
+
+ response.setMasterAddress("");
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<GetReplicaInfoResponseHeader> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<GetReplicaInfoResponseHeader> result = new ControllerResult<>(new GetReplicaInfoResponseHeader());
+ final GetReplicaInfoResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ // If exist broker metadata, just return metadata
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ response.setMasterAddress(replicasInfo.getMasterAddress());
+ response.setMasterEpoch(replicasInfo.getMasterEpoch());
+ response.setSyncStateSet(replicasInfo.getSyncStateSet());
+ response.setSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
Review Comment:
除了都返回INVALID_REQUEST，可能需要一些更加详细的错误码
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857059501
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
讨论方案: 让 Dledger 在 statemachine 恢复到 commit point 之后, 再通知 role change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] cserwen commented on a diff in pull request #4195: Dledger controller
Posted by GitBox <gi...@apache.org>.
cserwen commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r855788047
##########
common/src/main/java/org/apache/rocketmq/common/namesrv/NamesrvConfig.java:
##########
@@ -200,4 +221,68 @@ public boolean isEnableAllTopicList() {
public void setEnableAllTopicList(boolean enableAllTopicList) {
this.enableAllTopicList = enableAllTopicList;
}
+
+ public boolean isStartupController() {
+ return isStartupController;
+ }
+
+ public void setStartupController(boolean startupController) {
+ isStartupController = startupController;
+ }
+
+ public int getControllerThreadPoolNums() {
+ return controllerThreadPoolNums;
+ }
+
+ public void setControllerThreadPoolNums(int controllerThreadPoolNums) {
+ this.controllerThreadPoolNums = controllerThreadPoolNums;
+ }
+
+ public int getControllerRequestThreadPoolQueueCapacity() {
+ return controllerRequestThreadPoolQueueCapacity;
+ }
+
+ public void setControllerRequestThreadPoolQueueCapacity(int controllerRequestThreadPoolQueueCapacity) {
+ this.controllerRequestThreadPoolQueueCapacity = controllerRequestThreadPoolQueueCapacity;
+ }
+
+ public String getControllerDLegerGroup() {
+ return controllerDLegerGroup;
+ }
+
+ public void setControllerDLegerGroup(String controllerDLegerGroup) {
+ this.controllerDLegerGroup = controllerDLegerGroup;
+ }
+
+ public String getControllerDLegerPeers() {
+ return controllerDLegerPeers;
+ }
+
+ public void setControllerDLegerPeers(String controllerDLegerPeers) {
+ this.controllerDLegerPeers = controllerDLegerPeers;
+ }
+
+ public String getControllerDLegerSelfId() {
+ return controllerDLegerSelfId;
+ }
+
+ public void setControllerDLegerSelfId(String controllerDLegerSelfId) {
+ this.controllerDLegerSelfId = controllerDLegerSelfId;
+ }
+
+ public String getControllerStorePath() {
+ return controllerStorePath;
+ }
+
+ public void setControllerStorePath(String controllerStorePath) {
+ this.controllerStorePath = controllerStorePath;
+ }
+
+ public boolean isEnableElectUncleanMaster() {
+ return enableElectUncleanMaster;
+ }
+
+ public void setEnableElectUncleanMaster(boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ }
Review Comment:
@dugenkui03
The setter method is required, otherwise the configuration will not take effect.
https://github.com/apache/rocketmq/blob/bd3059e76a00b0496ba0e2e433f6266f2ceb5683/namesrv/src/main/java/org/apache/rocketmq/namesrv/NamesrvStartup.java#L93
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] RongtongJin commented on a diff in pull request #4195: Dledger controller
Posted by GitBox <gi...@apache.org>.
RongtongJin commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r855814522
##########
common/src/main/java/org/apache/rocketmq/common/protocol/header/namesrv/controller/ErrorCodes.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.common.protocol.header.namesrv.controller;
+
+/**
+ * @author hzh
+ * @email 642256541@qq.com
+ * @date 2022/4/16 20:01
+ */
Review Comment:
Do not leave author information
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/processor/ControllerRequestProcessor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.processor;
+
+import io.netty.channel.ChannelHandlerContext;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.remoting.common.RemotingHelper;
+import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_ALTER_SYNC_STATE_SET;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_ELECT_MASTER;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_GET_METADATA_INFO;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_GET_REPLICA_INFO;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_REGISTER_BROKER;
+
+/**
+ * Processor for controller request
+ */
+public class ControllerRequestProcessor implements NettyRequestProcessor {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final int WAIT_TIMEOUT_OUT = 10;
Review Comment:
Is the timeout here too long? Maybe 5 seconds better
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
Review Comment:
It is would be better to follow the rocketmq comment format
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
Will generating events based on the values in memory cause duplicate brokerId or epoch?
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/processor/ControllerRequestProcessor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.processor;
+
+import io.netty.channel.ChannelHandlerContext;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.remoting.common.RemotingHelper;
+import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_ALTER_SYNC_STATE_SET;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_ELECT_MASTER;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_GET_METADATA_INFO;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_GET_REPLICA_INFO;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_REGISTER_BROKER;
+
+/**
+ * Processor for controller request
+ */
+public class ControllerRequestProcessor implements NettyRequestProcessor {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final int WAIT_TIMEOUT_OUT = 10;
+ private final Controller controller;
+
+
+ public ControllerRequestProcessor(final Controller controller) {
+ this.controller = controller;
+ }
+
+ @Override
+ public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request) throws Exception {
+ if (ctx != null) {
+ log.debug("Receive request, {} {} {}",
+ request.getCode(),
+ RemotingHelper.parseChannelRemoteAddr(ctx.channel()),
+ request);
+ }
+ switch(request.getCode()) {
+ case CONTROLLER_ALTER_SYNC_STATE_SET : {
+ final AlterSyncStateSetRequestHeader controllerRequest = request.decodeCommandCustomHeader(AlterSyncStateSetRequestHeader.class);
+ final CompletableFuture<AlterSyncStateSetResponseHeader> future = this.controller.alterSyncStateSet(controllerRequest);
+ if (future != null) {
+ final AlterSyncStateSetResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_ELECT_MASTER: {
+ final ElectMasterRequestHeader controllerRequest = request.decodeCommandCustomHeader(ElectMasterRequestHeader.class);
+ final CompletableFuture<ElectMasterResponseHeader> future = this.controller.electMaster(controllerRequest);
+ if (future != null) {
+ final ElectMasterResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_REGISTER_BROKER: {
+ final RegisterBrokerRequestHeader controllerRequest = request.decodeCommandCustomHeader(RegisterBrokerRequestHeader.class);
+ final CompletableFuture<RegisterBrokerResponseHeader> future = this.controller.registerBroker(controllerRequest);
+ if (future != null) {
+ final RegisterBrokerResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_GET_REPLICA_INFO: {
+ final GetReplicaInfoRequestHeader controllerRequest = request.decodeCommandCustomHeader(GetReplicaInfoRequestHeader.class);
+ final CompletableFuture<GetReplicaInfoResponseHeader> future = this.controller.getReplicaInfo(controllerRequest);
+ if (future != null) {
+ final GetReplicaInfoResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_GET_METADATA_INFO: {
+ final GetMetaDataResponseHeader resp = this.controller.getControllerMetadata();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ default: {
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.QUERY_NOT_FOUND, null);
Review Comment:
QUERY_NOT_FOUND is not suitable here. It is usually used to not find consumer offset. IMO, it would be better to return RemotingSysResponseCode.REQUEST_CODE_NOT_SUPPORTED here.
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/processor/ControllerRequestProcessor.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.processor;
+
+import io.netty.channel.ChannelHandlerContext;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.ResponseCode;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.remoting.common.RemotingHelper;
+import org.apache.rocketmq.remoting.netty.NettyRequestProcessor;
+import org.apache.rocketmq.remoting.protocol.RemotingCommand;
+
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_ALTER_SYNC_STATE_SET;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_ELECT_MASTER;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_GET_METADATA_INFO;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_GET_REPLICA_INFO;
+import static org.apache.rocketmq.common.protocol.RequestCode.CONTROLLER_REGISTER_BROKER;
+
+/**
+ * Processor for controller request
+ */
+public class ControllerRequestProcessor implements NettyRequestProcessor {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final int WAIT_TIMEOUT_OUT = 10;
+ private final Controller controller;
+
+
+ public ControllerRequestProcessor(final Controller controller) {
+ this.controller = controller;
+ }
+
+ @Override
+ public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request) throws Exception {
+ if (ctx != null) {
+ log.debug("Receive request, {} {} {}",
+ request.getCode(),
+ RemotingHelper.parseChannelRemoteAddr(ctx.channel()),
+ request);
+ }
+ switch(request.getCode()) {
+ case CONTROLLER_ALTER_SYNC_STATE_SET : {
+ final AlterSyncStateSetRequestHeader controllerRequest = request.decodeCommandCustomHeader(AlterSyncStateSetRequestHeader.class);
+ final CompletableFuture<AlterSyncStateSetResponseHeader> future = this.controller.alterSyncStateSet(controllerRequest);
+ if (future != null) {
+ final AlterSyncStateSetResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_ELECT_MASTER: {
+ final ElectMasterRequestHeader controllerRequest = request.decodeCommandCustomHeader(ElectMasterRequestHeader.class);
+ final CompletableFuture<ElectMasterResponseHeader> future = this.controller.electMaster(controllerRequest);
+ if (future != null) {
+ final ElectMasterResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_REGISTER_BROKER: {
+ final RegisterBrokerRequestHeader controllerRequest = request.decodeCommandCustomHeader(RegisterBrokerRequestHeader.class);
+ final CompletableFuture<RegisterBrokerResponseHeader> future = this.controller.registerBroker(controllerRequest);
+ if (future != null) {
+ final RegisterBrokerResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_GET_REPLICA_INFO: {
+ final GetReplicaInfoRequestHeader controllerRequest = request.decodeCommandCustomHeader(GetReplicaInfoRequestHeader.class);
+ final CompletableFuture<GetReplicaInfoResponseHeader> future = this.controller.getReplicaInfo(controllerRequest);
+ if (future != null) {
+ final GetReplicaInfoResponseHeader resp = future.get(WAIT_TIMEOUT_OUT, TimeUnit.SECONDS);
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ break;
+ }
+ case CONTROLLER_GET_METADATA_INFO: {
+ final GetMetaDataResponseHeader resp = this.controller.getControllerMetadata();
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SUCCESS, resp);
+ }
+ default: {
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.QUERY_NOT_FOUND, null);
+ }
+ }
+ return RemotingCommand.createResponseCommandWithHeader(ResponseCode.SYSTEM_ERROR, null);
Review Comment:
Here you can refer to DefaultRequestProcessor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857059424
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/NamesrvController.java:
##########
@@ -82,24 +85,36 @@ public class NamesrvController {
private ExecutorService defaultExecutor;
private ExecutorService clientRequestExecutor;
+ private ExecutorService controllerRequestExecutor;
private BlockingQueue<Runnable> defaultThreadPoolQueue;
private BlockingQueue<Runnable> clientRequestThreadPoolQueue;
+ private BlockingQueue<Runnable> controllerRequestThreadPoolQueue;
private Configuration configuration;
private FileWatchService fileWatchService;
+ private Controller controller;
+
public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig) {
this(namesrvConfig, nettyServerConfig, new NettyClientConfig());
}
- public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig, NettyClientConfig nettyClientConfig) {
+ public NamesrvController(NamesrvConfig namesrvConfig, NettyServerConfig nettyServerConfig,
+ NettyClientConfig nettyClientConfig) {
this.namesrvConfig = namesrvConfig;
this.nettyServerConfig = nettyServerConfig;
this.nettyClientConfig = nettyClientConfig;
this.kvConfigManager = new KVConfigManager(this);
this.brokerHousekeepingService = new BrokerHousekeepingService(this);
- this.routeInfoManager = new RouteInfoManager(namesrvConfig, this);
+ if (namesrvConfig.isStartupController()) {
+ final DLedgerConfig config = new DLedgerConfig();
+ config.setGroup(namesrvConfig.getControllerDLegerGroup());
+ config.setPeers(namesrvConfig.getControllerDLegerPeers());
+ config.setSelfId(namesrvConfig.getControllerDLegerSelfId());
+ this.controller = new DledgerController(config, namesrvConfig.isEnableElectUncleanMaster());
Review Comment:
讨论方案: 新增一个 DledgerControllerConfig
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.common.ServiceThread;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final DLedgerConfig dLedgerConfig;
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final DLedgerConfig dLedgerConfig, final boolean isEnableElectUncleanMaster) {
+ this.dLedgerConfig = dLedgerConfig;
+
+ this.eventSerializer = new EventSerializer();
+
+ this.scheduler = new EventScheduler();
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(isEnableElectUncleanMaster);
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ AlterSyncStateSetRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject alterSyncStateSet request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request), true);
+ }
+
+ @Override
+ public CompletableFuture<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject electMaster request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RegisterBrokerResponseHeader> registerBroker(RegisterBrokerRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject registerBroker request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<GetReplicaInfoResponseHeader> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject getReplicaInfo request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public GetMetaDataResponseHeader getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr());
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+
+ }
+
+ public <T> CompletableFuture<T> appendEvent(final String name, final Supplier<ControllerResult<T>> supplier,
+ boolean isWriteEvent) {
+ if (isStopped()) {
+ return null;
+ }
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<T> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
Review Comment:
讨论方案: 先开一个配置(isProcessReadEvet = {ReadEvent 是否要走日志}). 后续可以为 Dledger 新增 Read-Index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#issuecomment-1107516899
@RongtongJin Thanks a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857054556
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.common.ServiceThread;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final DLedgerConfig dLedgerConfig;
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final DLedgerConfig dLedgerConfig, final boolean isEnableElectUncleanMaster) {
+ this.dLedgerConfig = dLedgerConfig;
+
+ this.eventSerializer = new EventSerializer();
+
+ this.scheduler = new EventScheduler();
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(isEnableElectUncleanMaster);
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ AlterSyncStateSetRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject alterSyncStateSet request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request), true);
+ }
+
+ @Override
+ public CompletableFuture<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject electMaster request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RegisterBrokerResponseHeader> registerBroker(RegisterBrokerRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject registerBroker request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<GetReplicaInfoResponseHeader> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject getReplicaInfo request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public GetMetaDataResponseHeader getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr());
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+
+ }
+
+ public <T> CompletableFuture<T> appendEvent(final String name, final Supplier<ControllerResult<T>> supplier,
+ boolean isWriteEvent) {
+ if (isStopped()) {
+ return null;
+ }
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<T> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
Review Comment:
嗯, 这个还有待讨论.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r857059326
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/impl/DledgerController.java:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.impl;
+
+import io.openmessaging.storage.dledger.AppendFuture;
+import io.openmessaging.storage.dledger.DLedgerConfig;
+import io.openmessaging.storage.dledger.DLedgerLeaderElector;
+import io.openmessaging.storage.dledger.DLedgerServer;
+import io.openmessaging.storage.dledger.MemberState;
+import io.openmessaging.storage.dledger.protocol.AppendEntryRequest;
+import io.openmessaging.storage.dledger.protocol.AppendEntryResponse;
+import io.openmessaging.storage.dledger.protocol.BatchAppendEntryRequest;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetMetaDataResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.Controller;
+import org.apache.rocketmq.namesrv.controller.manager.ReplicasInfoManager;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventSerializer;
+import org.apache.rocketmq.remoting.common.ServiceThread;
+
+/**
+ * The implementation of controller, based on dledger (raft).
+ */
+public class DledgerController implements Controller {
+
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private final DLedgerServer dLedgerServer;
+ private final DLedgerConfig dLedgerConfig;
+ private final ReplicasInfoManager replicasInfoManager;
+ private final EventScheduler scheduler;
+ private final EventSerializer eventSerializer;
+ private final RoleChangeHandler roleHandler;
+ private final DledgerControllerStateMachine statemachine;
+ private volatile boolean isScheduling = false;
+
+ public DledgerController(final DLedgerConfig dLedgerConfig, final boolean isEnableElectUncleanMaster) {
+ this.dLedgerConfig = dLedgerConfig;
+
+ this.eventSerializer = new EventSerializer();
+
+ this.scheduler = new EventScheduler();
+ this.roleHandler = new RoleChangeHandler(dLedgerConfig.getSelfId());
+ this.replicasInfoManager = new ReplicasInfoManager(isEnableElectUncleanMaster);
+ this.statemachine = new DledgerControllerStateMachine(replicasInfoManager, this.eventSerializer, dLedgerConfig.getSelfId());
+
+ // Register statemachine and role handler.
+ this.dLedgerServer = new DLedgerServer(dLedgerConfig);
+ this.dLedgerServer.registerStateMachine(this.statemachine);
+ this.dLedgerServer.getdLedgerLeaderElector().addRoleChangeHandler(this.roleHandler);
+ }
+
+ @Override
+ public void startup() {
+ this.dLedgerServer.startup();
+ }
+
+ @Override
+ public void shutdown() {
+ this.dLedgerServer.shutdown();
+ }
+
+ @Override
+ public void startScheduling() {
+ if (!this.isScheduling) {
+ log.info("Start scheduling controller events");
+ this.isScheduling = true;
+ this.scheduler.start();
+ }
+ }
+
+ @Override
+ public void stopScheduling() {
+ if (this.isScheduling) {
+ log.info("Stop scheduling controller events");
+ this.isScheduling = false;
+ this.scheduler.shutdown(true);
+ }
+ }
+
+ @Override
+ public CompletableFuture<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ AlterSyncStateSetRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject alterSyncStateSet request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("alterSyncStateSet",
+ () -> this.replicasInfoManager.alterSyncStateSet(request), true);
+ }
+
+ @Override
+ public CompletableFuture<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject electMaster request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("electMaster",
+ () -> this.replicasInfoManager.electMaster(request), true);
+ }
+
+ @Override
+ public CompletableFuture<RegisterBrokerResponseHeader> registerBroker(RegisterBrokerRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject registerBroker request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("registerBroker",
+ () -> this.replicasInfoManager.registerBroker(request), true);
+ }
+
+ @Override
+ public CompletableFuture<GetReplicaInfoResponseHeader> getReplicaInfo(final GetReplicaInfoRequestHeader request) {
+ if (!this.roleHandler.isLeaderState()) {
+ log.warn("Current controller {} is not leader, reject getReplicaInfo request", this.dLedgerConfig.getSelfId());
+ return null;
+ }
+ return this.scheduler.appendEvent("getReplicaInfo",
+ () -> this.replicasInfoManager.getReplicaInfo(request), false);
+ }
+
+ @Override
+ public GetMetaDataResponseHeader getControllerMetadata() {
+ final MemberState state = getMemberState();
+ return new GetMetaDataResponseHeader(state.getLeaderId(), state.getLeaderAddr());
+ }
+
+ /**
+ * Event scheduler, schedule event handler from event queue
+ */
+ class EventScheduler extends ServiceThread {
+ private final BlockingQueue<EventHandler> eventQueue;
+
+ public EventScheduler() {
+ this.eventQueue = new LinkedBlockingQueue<>(1024);
+ }
+
+ @Override
+ public String getServiceName() {
+ return EventScheduler.class.getName();
+ }
+
+ @Override
+ public void run() {
+ log.info("Start event scheduler.");
+ while (!isStopped()) {
+ EventHandler handler;
+ try {
+ handler = this.eventQueue.poll(5, TimeUnit.SECONDS);
+ } catch (final InterruptedException e) {
+ continue;
+ }
+ try {
+ if (handler != null) {
+ handler.run();
+ }
+ } catch (final Throwable e) {
+ handler.handleException(e);
+ }
+ }
+
+ }
+
+ public <T> CompletableFuture<T> appendEvent(final String name, final Supplier<ControllerResult<T>> supplier,
+ boolean isWriteEvent) {
+ if (isStopped()) {
+ return null;
+ }
+ final EventHandler<T> event = new ControllerEventHandler<>(name, supplier, isWriteEvent);
+ int tryTimes = 0;
+ while (true) {
+ try {
+ if (!this.eventQueue.offer(event, 5, TimeUnit.SECONDS)) {
+ continue;
+ }
+ return event.future();
+ } catch (final InterruptedException e) {
+ log.error("Error happen in EventScheduler when append event", e);
+ tryTimes++;
+ if (tryTimes > 3) {
+ return null;
+ }
+ }
+ }
+ }
+ }
+
+ /**
+ * Event handler, get events from supplier, and append events to dledger
+ */
+ class ControllerEventHandler<T> implements EventHandler<T> {
+ private final String name;
+ private final Supplier<ControllerResult<T>> supplier;
+ private final CompletableFuture<T> future;
+ private final boolean isWriteEvent;
+
+ ControllerEventHandler(final String name, final Supplier<ControllerResult<T>> supplier,
+ final boolean isWriteEvent) {
+ this.name = name;
+ this.supplier = supplier;
+ this.future = new CompletableFuture<>();
+ this.isWriteEvent = isWriteEvent;
+ }
+
+ @Override
+ public void run() throws Throwable {
+ final ControllerResult<T> result = this.supplier.get();
+ log.info("Event queue run event {}, get the result {}", this.name, result);
+ boolean appendSuccess = true;
+ if (this.isWriteEvent) {
+ final List<EventMessage> events = result.getEvents();
+ final List<byte[]> eventBytes = new ArrayList<>(events.size());
+ for (final EventMessage event : events) {
+ if (event != null) {
+ final byte[] data = DledgerController.this.eventSerializer.serialize(event);
+ if (data != null && data.length > 0) {
+ eventBytes.add(data);
+ }
+ }
+ }
+ // Append events to dledger
+ if (!eventBytes.isEmpty()) {
+ final BatchAppendEntryRequest request = new BatchAppendEntryRequest();
+ request.setBatchMsgs(eventBytes);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
+ } else {
+ // Now the dledger don't have the function of Read-Index or Lease-Read,
+ // So we still need to propose an empty request to dledger.
+ final AppendEntryRequest request = new AppendEntryRequest();
+ request.setBody(new byte[0]);
+ appendSuccess = appendToDledgerAndWait(request);
+ }
Review Comment:
解决方案: 先开一个配置(isProcessReadEvet = {ReadEvent 是否要走日志}). 后续可以为 Dledger 新增 Read-Index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [rocketmq] hzh0425 commented on a diff in pull request #4195: [Summer of Code] Dledger controller
Posted by GitBox <gi...@apache.org>.
hzh0425 commented on code in PR #4195:
URL: https://github.com/apache/rocketmq/pull/4195#discussion_r856146275
##########
namesrv/src/main/java/org/apache/rocketmq/namesrv/controller/manager/ReplicasInfoManager.java:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.rocketmq.namesrv.controller.manager;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.rocketmq.common.constant.LoggerName;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.AlterSyncStateSetResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ElectMasterResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.ErrorCodes;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.GetReplicaInfoResponseHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerRequestHeader;
+import org.apache.rocketmq.common.protocol.header.namesrv.controller.RegisterBrokerResponseHeader;
+import org.apache.rocketmq.logging.InternalLogger;
+import org.apache.rocketmq.logging.InternalLoggerFactory;
+import org.apache.rocketmq.namesrv.controller.manager.event.AlterSyncStateSetEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ApplyBrokerIdEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.ControllerResult;
+import org.apache.rocketmq.namesrv.controller.manager.event.ElectMasterEvent;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventMessage;
+import org.apache.rocketmq.namesrv.controller.manager.event.EventType;
+
+/**
+ * The manager that manages the replicas info for all brokers.
+ * We can think of this class as the controller's memory state machine
+ * It should be noted that this class is not thread safe,
+ * and the upper layer needs to ensure that it can be called sequentially
+ */
+public class ReplicasInfoManager {
+ private static final InternalLogger log = InternalLoggerFactory.getLogger(LoggerName.CONTROLLER_LOGGER_NAME);
+ private static final Long LEADER_ID = 0L;
+ private final boolean enableElectUncleanMaster;
+ private final Map<String/* brokerName */, BrokerIdInfo> replicaInfoTable;
+ private final Map<String/* brokerName */, InSyncReplicasInfo> inSyncReplicasInfoTable;
+
+ public ReplicasInfoManager(final boolean enableElectUncleanMaster) {
+ this.enableElectUncleanMaster = enableElectUncleanMaster;
+ this.replicaInfoTable = new HashMap<>();
+ this.inSyncReplicasInfoTable = new HashMap<>();
+ }
+
+ /******************************** The following methods don't update statemachine, triggered by controller leader ********************************/
+
+ public ControllerResult<AlterSyncStateSetResponseHeader> alterSyncStateSet(
+ final AlterSyncStateSetRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<AlterSyncStateSetResponseHeader> result = new ControllerResult<>(new AlterSyncStateSetResponseHeader());
+ final AlterSyncStateSetResponseHeader response = result.getResponse();
+
+ if (isContainsBroker(brokerName)) {
+ final Set<String> newSyncStateSet = request.getNewSyncStateSet();
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Check master
+ if (!replicasInfo.getMasterAddress().equals(request.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the current leader is:{}, not {}",
+ replicasInfo.getMasterAddress(), request.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Check master epoch
+ if (request.getMasterEpoch() != replicasInfo.getMasterEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current master epoch is:{}, not {}",
+ replicasInfo.getMasterEpoch(), request.getMasterEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_LEADER_EPOCH.getCode());
+ return result;
+ }
+
+ // Check syncStateSet epoch
+ if (request.getSyncStateSetEpoch() != replicasInfo.getSyncStateSetEpoch()) {
+ log.info("Rejecting alter syncStateSet request because the current syncStateSet epoch is:{}, not {}",
+ replicasInfo.getSyncStateSetEpoch(), request.getSyncStateSetEpoch());
+ response.setErrorCode(ErrorCodes.FENCED_SYNC_STATE_SET_EPOCH.getCode());
+ return result;
+ }
+
+ // Check newSyncStateSet correctness
+ for (String replicas : newSyncStateSet) {
+ if (!brokerIdTable.containsKey(replicas)) {
+ log.info("Rejecting alter syncStateSet request because the replicas {} don't exist", replicas);
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+ // todo: check whether the replicas is active
+ }
+ if (!newSyncStateSet.contains(replicasInfo.getMasterAddress())) {
+ log.info("Rejecting alter syncStateSet request because the newSyncStateSet don't contains origin leader {}", replicasInfo.getMasterAddress());
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ // Generate event
+ response.setNewSyncStateSetEpoch(replicasInfo.getSyncStateSetEpoch() + 1);
+ response.setNewSyncStateSet(newSyncStateSet);
+ final AlterSyncStateSetEvent event = new AlterSyncStateSetEvent(brokerName, newSyncStateSet);
+ result.addEvent(event);
+
+ return result;
+ }
+ response.setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<ElectMasterResponseHeader> electMaster(final ElectMasterRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final ControllerResult<ElectMasterResponseHeader> result = new ControllerResult<>(new ElectMasterResponseHeader());
+ final ElectMasterResponseHeader response = result.getResponse();
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final Set<String> syncStateSet = replicasInfo.getSyncStateSet();
+ if (syncStateSet.size() > 1) {
+ for (String replicas : syncStateSet) {
+ if (replicas.equals(replicasInfo.getMasterAddress())) {
+ continue;
+ }
+ // todo: check whether the replicas is active
+ response.setNewMasterAddress(replicas);
+ response.setMasterEpoch(replicasInfo.getMasterEpoch() + 1);
+
+ final ElectMasterEvent event = new ElectMasterEvent(brokerName, replicas);
+ result.addEvent(event);
+ return result;
+ }
+ }
+ // If elect failed, we still need to apply an ElectMasterEvent to tell the statemachine
+ // that the master was shutdown and no new master was elected.
+ final ElectMasterEvent event = new ElectMasterEvent(false, brokerName);
+ result.addEvent(event);
+ response.setErrorCode(ErrorCodes.MASTER_NOT_AVAILABLE.getCode());
+ return result;
+ }
+ result.getResponse().setErrorCode(ErrorCodes.INVALID_REQUEST.getCode());
+ return result;
+ }
+
+ public ControllerResult<RegisterBrokerResponseHeader> registerBroker(final RegisterBrokerRequestHeader request) {
+ final String brokerName = request.getBrokerName();
+ final String brokerAddress = request.getBrokerAddress();
+ final ControllerResult<RegisterBrokerResponseHeader> result = new ControllerResult<>(new RegisterBrokerResponseHeader());
+ final RegisterBrokerResponseHeader response = result.getResponse();
+ boolean canBeElectedAsMaster;
+ if (isContainsBroker(brokerName)) {
+ final InSyncReplicasInfo replicasInfo = this.inSyncReplicasInfoTable.get(brokerName);
+ final BrokerIdInfo brokerInfo = this.replicaInfoTable.get(brokerName);
+ final HashMap<String, Long> brokerIdTable = brokerInfo.getBrokerIdTable();
+
+ // Get brokerId.
+ long brokerId;
+ if (!brokerIdTable.containsKey(brokerAddress)) {
+ // If this broker replicas is first time come online, we need to apply a new id for this replicas.
+ brokerId = brokerInfo.newBrokerId();
+ final ApplyBrokerIdEvent applyIdEvent = new ApplyBrokerIdEvent(request.getBrokerName(),
+ brokerAddress, brokerId);
+ result.addEvent(applyIdEvent);
Review Comment:
Actually not, because eventScheduler is scheduled in fifo order.
The next eventHandler will be dispatched only after the events generated by the previous eventHandler are appended to the dledger and applied to the state machine.
So it doesn't happen as you said.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@rocketmq.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org