You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/02/05 18:33:49 UTC
[GitHub] [kafka] cmccabe opened a new pull request #10070: KAFKA-12276: Add the quorum controller code
cmccabe opened a new pull request #10070:
URL: https://github.com/apache/kafka/pull/10070
The quorum controller stores metadata in the KIP-500 metadata log, not in Apache ZooKeeper. Each controller node is a voter in the metadata quorum. The leader of the quorum is the active controller, which processes write requests. The followers are standby controllers, which replay the operations written to the log. If the active controller goes away, a standby controller can take its place.
Like the ZooKeeper-based controller, the quorum controller is based on an event queue backed by a single-threaded executor. However, unlike the ZK-based controller, the quorum controller can have multiple operations in flight-- it does not need to wait for one operation to be finished before starting another. Therefore, calls into the QuorumController return CompleteableFuture objects which are completed with either a result or an error when the operation is done. The QuorumController will also time out operations that have been sitting on the queue too long without being processed. In this case, the future is completed with a TimeoutException.
The controller uses timeline data structures to store multiple "versions" of its in-memory state simultaneously. "Read operations" read the latest committed state -- essentially, they are reading from a slightly older in-memory snapshot of the state. "Write operations" read and write the latest state. However, we can not return a successful result for a write operation until its state has been committed to the log. Therefore, if a client receives an RPC response, it knows that the requested operation has been performed, and can not be undone by a controller failover.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579496903
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
Review comment:
Yes, the alterIsr doesn't change leader, but generates a PartitionChangeRecord. On replaying this record, the code following code bumps on leaderEpoch?
` PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573256612
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
Review comment:
I wanted to standardize on shutDown. I will fix the inconsistency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577961646
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,916 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import static java.util.concurrent.TimeUnit.MICROSECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = NANOSECONDS.convert(18, TimeUnit.SECONDS);
+ private ControllerMetrics controllerMetrics = null;
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public Builder setMetrics(ControllerMetrics controllerMetrics) {
+ this.controllerMetrics = controllerMetrics;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ if (controllerMetrics == null) {
+ controllerMetrics = (ControllerMetrics) Class.forName(
+ "org.apache.kafka.controller.MockControllerMetrics").getConstructor().newInstance();
Review comment:
`MockControllerMetrics` is a test class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577167941
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef.ConfigKey;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource.Type;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.APPEND;
+
+public class ConfigurationControlManager {
+ private final Logger log;
+ private final SnapshotRegistry snapshotRegistry;
+ private final Map<ConfigResource.Type, ConfigDef> configDefs;
+ private final TimelineHashMap<ConfigResource, TimelineHashMap<String, String>> configData;
+
+ ConfigurationControlManager(LogContext logContext,
+ SnapshotRegistry snapshotRegistry,
+ Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.log = logContext.logger(ConfigurationControlManager.class);
+ this.snapshotRegistry = snapshotRegistry;
+ this.configDefs = configDefs;
+ this.configData = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Determine the result of applying a batch of incremental configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * Note that there can only be one result per ConfigResource. So if you try to modify
+ * several keys and one modification fails, the whole ConfigKey fails and nothing gets
+ * changed.
+ *
+ * @param configChanges Maps each resource to a map from config keys to
+ * operation data.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, Entry<OpType, String>>> resourceEntry :
+ configChanges.entrySet()) {
+ incrementalAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void incrementalAlterConfigResource(ConfigResource configResource,
+ Map<String, Entry<OpType, String>> keysToOps,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ for (Entry<String, Entry<OpType, String>> keysToOpsEntry : keysToOps.entrySet()) {
+ String key = keysToOpsEntry.getKey();
+ String currentValue = null;
+ TimelineHashMap<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs != null) {
+ currentValue = currentConfigs.get(key);
+ }
+ String newValue = currentValue;
+ Entry<OpType, String> opTypeAndNewValue = keysToOpsEntry.getValue();
+ OpType opType = opTypeAndNewValue.getKey();
+ String opValue = opTypeAndNewValue.getValue();
+ switch (opType) {
+ case SET:
+ newValue = opValue;
+ break;
+ case DELETE:
+ if (opValue != null) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_REQUEST, "A DELETE op was given with a " +
+ "non-null value."));
+ return;
+ }
+ newValue = null;
+ break;
+ case APPEND:
+ case SUBTRACT:
+ if (!isSplittable(configResource.type(), key)) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_CONFIG, "Can't " + opType + " to " +
+ "key " + key + " because its type is not LIST."));
+ return;
+ }
+ List<String> newValueParts = getParts(newValue, key, configResource);
+ if (opType == APPEND) {
+ if (!newValueParts.contains(opValue)) {
+ newValueParts.add(opValue);
+ }
+ newValue = String.join(",", newValueParts);
+ } else if (newValueParts.remove(opValue)) {
+ newValue = String.join(",", newValueParts);
+ }
+ break;
+ }
+ if (!Objects.equals(currentValue, newValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ /**
+ * Determine the result of applying a batch of legacy configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * @param newConfigs The new configurations to install for each resource.
+ * All existing configurations will be overwritten.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, String>> resourceEntry :
+ newConfigs.entrySet()) {
+ legacyAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void legacyAlterConfigResource(ConfigResource configResource,
+ Map<String, String> newConfigs,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ Map<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs == null) {
+ currentConfigs = Collections.emptyMap();
+ }
+ for (Entry<String, String> entry : newConfigs.entrySet()) {
+ String key = entry.getKey();
+ String newValue = entry.getValue();
+ String currentValue = currentConfigs.get(key);
+ if (!Objects.equals(newValue, currentValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ for (String key : currentConfigs.keySet()) {
+ if (!newConfigs.containsKey(key)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(null), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ private List<String> getParts(String value, String key, ConfigResource configResource) {
+ if (value == null) {
+ value = getConfigValueDefault(configResource.type(), key);
+ }
+ List<String> parts = new ArrayList<>();
+ if (value == null) {
+ return parts;
+ }
+ String[] splitValues = value.split(",");
+ for (String splitValue : splitValues) {
+ if (!splitValue.isEmpty()) {
+ parts.add(splitValue);
+ }
+ }
+ return parts;
+ }
+
+ static ApiError checkConfigResource(ConfigResource configResource) {
+ switch (configResource.type()) {
+ case BROKER_LOGGER:
+ // We do not handle resources of type BROKER_LOGGER in
Review comment:
This has to be handled by the individual brokers. It's not persisted anywhere currently (currently it is not stored in ZK I believe)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579411851
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
Review comment:
Currently, for leader initiated AlterIsr request, the controller doesn't bump up the leader epoch. If we change that, it will slightly increase unavailability since all clients have to refresh the metadata in this case.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param alwaysBumpLeaderEpoch True if we should generate a record that
+ * always increments the leader epoch.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, boolean alwaysBumpLeaderEpoch,
Review comment:
Currently, for controller initiated ISR change (controlled shutdown or hard failure), we always bump up the leader epoch. Also, the name alwaysBumpLeaderEpoch is a bit weird since the code in handleNodeDeactivated() doesn't directly bump up leader epoch.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param alwaysBumpLeaderEpoch True if we should generate a record that
+ * always increments the leader epoch.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, boolean alwaysBumpLeaderEpoch,
+ List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ PartitionChangeRecord record = new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id);
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the ISR.
+ if (alwaysBumpLeaderEpoch || record.leader() != -1) {
+ // The partition is now leaderless, so set its leader to -1.
+ record.setLeader(-1);
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ }
+ } else {
+ record.setIsr(Replicas.toList(newIsr));
+ if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ int newLeader = chooseNewLeader(partition, newIsr, false);
+ record.setLeader(newLeader);
+ } else if (alwaysBumpLeaderEpoch) {
+ record.setLeader(partition.leader);
+ }
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ }
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setLeader(brokerId), (short) 0));
+ }
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ ElectLeadersResponseData response = new ElectLeadersResponseData();
+ for (TopicPartitions topic : request.topicPartitions()) {
+ ReplicaElectionResult topicResults =
+ new ReplicaElectionResult().setTopic(topic.topic());
+ response.replicaElectionResults().add(topicResults);
+ for (int partitionId : topic.partitions()) {
+ ApiError error = electLeader(topic.topic(), partitionId, unclean, records);
+ topicResults.partitionResult().add(new PartitionResult().
+ setPartitionId(partitionId).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ static boolean electionIsUnclean(byte electionType) {
+ ElectionType type;
+ try {
+ type = ElectionType.valueOf(electionType);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidRequestException("Unknown election type " + (int) electionType);
+ }
+ return type == ElectionType.UNCLEAN;
+ }
+
+ ApiError electLeader(String topic, int partitionId, boolean unclean,
+ List<ApiMessageAndVersion> records) {
+ Uuid topicId = topicsByName.get(topic);
+ if (topicId == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic as " + topic);
+ }
+ TopicControlInfo topicInfo = topics.get(topicId);
+ if (topicInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic id as " + topicId);
+ }
+ PartitionControlInfo partitionInfo = topicInfo.parts.get(partitionId);
+ if (partitionInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such partition as " + topic + "-" + partitionId);
+ }
+ int newLeader = chooseNewLeader(partitionInfo, partitionInfo.isr, unclean);
+ if (newLeader < 0) {
+ // If we can't find any leader for the partition, return an error.
+ return new ApiError(Errors.LEADER_NOT_AVAILABLE,
+ "Unable to find any leader for the partition.");
+ }
+ if (newLeader == partitionInfo.leader) {
+ // If the new leader we picked is the same as the current leader, there is
+ // nothing to do.
+ return ApiError.NONE;
+ }
+ if (partitionInfo.leader != -1 && newLeader != partitionInfo.preferredReplica()) {
Review comment:
Hmm, it seems that we should only do `newLeader != partitionInfo.preferredReplica()` if this is a preferred leader election.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
Review comment:
Hmm, does Integer.MIN_VALUE have any special meaning? If so, could we use a more intuitive constant?
##########
File path: core/src/main/scala/kafka/server/BrokerServer.scala
##########
@@ -397,6 +397,9 @@ class BrokerServer(
info("shutting down")
if (config.controlledShutdownEnable) {
+ // Shut down the broker metadata listener, so that we don't get added to any
+ // more ISRs.
+ brokerMetadataListener.beginShutdown()
Review comment:
If we do this, does `brokerMetadataListener.close() `still need to call `beginShutdown()`.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param alwaysBumpLeaderEpoch True if we should generate a record that
+ * always increments the leader epoch.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, boolean alwaysBumpLeaderEpoch,
+ List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ PartitionChangeRecord record = new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id);
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the ISR.
+ if (alwaysBumpLeaderEpoch || record.leader() != -1) {
Review comment:
Hmm, if the leader is already -1 and we can't change ISR, there is no need to generate a new PartitionChangeRecord just to bump up the leader epoch. It won't help controlled shutdown since there is already no leader.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573240204
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
Review comment:
Sorry, was asking if it should be public -- but I assume not. Was just checking. It's fine now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577919831
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
Review comment:
The heartbeat manager is special because it stores soft state which is not in the metadata log (when each broker last heartbeated, for example).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577916633
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
Review comment:
This got refactored, hopefully the new version is clearer. The new version avoids the MAX_LONG / 0 hack and other ugliness that was in the initial version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573235373
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
Review comment:
Hmm, are you suggesting that it should be public? I'd rather not make this public because it's only accessed from within the controller package
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
Review comment:
Hmm, are you suggesting that it should be public? I'd rather not make this public because it's only accessed from within the controller package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577117523
##########
File path: metadata/src/main/java/org/apache/kafka/metalog/LocalLogManager.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.metalog;
+
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+
+/**
+ * The LocalLogManager is a test implementation that relies on the contents of memory.
Review comment:
Yes, it is is. I will move it to the test directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577921275
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
Review comment:
this comes out of the Raft layer and is invoked when the Raft election has succeeded and produced a new leader node.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579538752
##########
File path: metadata/src/main/resources/common/metadata/PartitionRecord.json
##########
@@ -34,6 +34,8 @@
{ "name": "Leader", "type": "int32", "versions": "0+", "default": "-1",
"about": "The lead replica, or -1 if there is no leader." },
{ "name": "LeaderEpoch", "type": "int32", "versions": "0+", "default": "-1",
- "about": "An epoch that gets incremented each time we change the ISR." }
+ "about": "An epoch that gets incremented each time we change the ISR." },
+ { "name": "PartitionEpoch", "type": "int32", "versions": "0+", "default": "-1",
Review comment:
Could you file a separate jira to follow up on PartitionEpoch post 2.8?
##########
File path: metadata/src/main/resources/common/metadata/PartitionRecord.json
##########
@@ -34,6 +34,8 @@
{ "name": "Leader", "type": "int32", "versions": "0+", "default": "-1",
"about": "The lead replica, or -1 if there is no leader." },
{ "name": "LeaderEpoch", "type": "int32", "versions": "0+", "default": "-1",
- "about": "An epoch that gets incremented each time we change the ISR." }
+ "about": "An epoch that gets incremented each time we change the ISR." },
+ { "name": "PartitionEpoch", "type": "int32", "versions": "0+", "default": "-1",
Review comment:
Could you file a separate jira to follow up on PartitionEpoch post 2.8?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ /**
+ * A special value used to represent the leader for a partition with no leader.
+ */
+ public static final int NO_LEADER = -1;
+
+ /**
+ * A special value used to represent a PartitionChangeRecord that does not change the
+ * partition leader.
+ */
+ public static final int NO_LEADER_CHANGE = -2;
+
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == NO_LEADER_CHANGE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ boolean hasLeader() {
+ return leader != NO_LEADER;
+ }
+
+ int preferredReplica() {
+ return replicas.length == 0 ? NO_LEADER : replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, NO_LEADER, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ continue;
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ continue;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
Review comment:
Hmm, we should set the leader to NO_LEADER_CHANGE, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577187004
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
+ isFenced = true;
+ shouldShutdown = true;
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
Review comment:
I will improve the logging a bit here. I agree that we should log when a broker is told it can shut down or be (un) fenced, since those are major events.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578628736
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request,
+ long brokerEpoch,
+ FeatureMapAndEpoch finalizedFeatures) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ Optional<VersionRange> finalized = finalizedFeatures.map().get(feature.name());
+ if (finalized.isPresent()) {
+ if (!finalized.get().contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
Review comment:
Good catch. This has been superseded by `ReplicationControlManager#unregsiterBroker`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579464755
##########
File path: core/src/main/scala/kafka/server/BrokerServer.scala
##########
@@ -397,6 +397,9 @@ class BrokerServer(
info("shutting down")
if (config.controlledShutdownEnable) {
+ // Shut down the broker metadata listener, so that we don't get added to any
+ // more ISRs.
+ brokerMetadataListener.beginShutdown()
Review comment:
It does need to be because there are some paths through the code that don't go through here. In general calling `beginShutdown` or `close` multiple times is harmless-- only the first time has an effect.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577922831
##########
File path: metadata/src/main/java/org/apache/kafka/controller/Controller.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+
+public interface Controller extends AutoCloseable {
+ /**
+ * Change partition ISRs.
+ *
+ * @param request The AlterIsrRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request);
+
+ /**
+ * Create a batch of topics.
+ *
+ * @param request The CreateTopicsRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request);
+
+ /**
+ * Decommission a broker.
+ *
+ * @param brokerId The broker id to decommission.
+ *
+ * @return A future that is completed successfully hwne the broker is
+ * decommissioned, or if it is not registered in the first place.
+ */
+ CompletableFuture<Void> decommissionBroker(int brokerId);
+
+ /**
+ * Describe the current configuration of various resources.
+ *
+ * @param resources A map from resources to the collection of config keys that we
+ * want to describe for each. If the collection is empty, then
+ * all configuration keys will be described.
+ *
+ * @return
+ */
+ CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources);
+
+ /**
+ * Elect new partition leaders.
+ *
+ * @param request The request.
+ *
+ * @return A future yielding the elect leaders response.
+ */
+ CompletableFuture<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request);
+
+ /**
+ * Get the current finalized feature ranges for each feature.
+ *
+ * @return A future yielding the feature ranges.
+ */
+ CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures();
+
+ /**
+ * Perform some incremental configuration changes.
+ *
+ * @param configChanges The changes.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
+ */
+ CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Map.Entry<AlterConfigOp.OpType, String>>> configChanges,
+ boolean validateOnly);
+
+ /**
+ * Perform some configuration changes using the legacy API.
+ *
+ * @param newConfigs The new configuration maps to apply.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
+ */
+ CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly);
+
+ /**
+ * Process a heartbeat from a broker.
+ *
+ * @param request The broker heartbeat request.
+ *
+ * @return A future yielding a heartbeat reply.
+ */
+ CompletableFuture<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request);
+
+ /**
+ * Attempt to register the given broker.
+ *
+ * @param request The registration request.
+ *
+ * @return A future yielding a registration reply.
+ */
+ CompletableFuture<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request);
+
+ /**
+ * Wait for the given number of brokers to be registered and unfenced.
+ * This is for testing.
+ *
+ * @param minBrokers The minimum number of brokers to wait for.
+ * @return A future which is completed when the given number of brokers
+ * is reached.
+ */
+ CompletableFuture<Void> waitForReadyBrokers(int minBrokers);
+
+ /**
+ * Perform some client quota changes
+ *
+ * @param quotaAlterations The list of quotas to alter
+ * @param validateOnly True if we should validate the changes but not apply them.
+ * @return A future yielding a map of quota entities to error results.
+ */
+ CompletableFuture<Map<ClientQuotaEntity, ApiError>> alterClientQuotas(
+ Collection<ClientQuotaAlteration> quotaAlterations, boolean validateOnly
+ );
+
+ /**
+ * Begin shutting down, but don't block. You must still call close to clean up all
+ * resources.
+ */
+ void beginShutdown();
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1.
+ */
+ long curClaimEpoch();
+
+ /**
+ * Blocks until we have shut down and freed all resources.
+ */
+ void close() throws InterruptedException;
Review comment:
The name close comes from `AutoCloseable`, which makes some of the tests nicer to write (because we can use the Java try-with-resources syntax).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#issuecomment-782521288
Thanks, @junrao ! :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577148107
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
Review comment:
If an exception is thrown here we will end up in `handleEventException`. Since the exception won't be a subclass of `ApiException` we will resign as controller and return an `UnknownServerException`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577920610
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokersToIsrs.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+
+
+/**
+ * Associates brokers with their in-sync partitions.
+ *
+ * This is useful when we need to remove a broker from all the ISRs, or move all leaders
+ * away from a broker.
+ *
+ * We also track all the partitions that currently have no leader.
+ *
+ * The core data structure is a map from broker IDs to topic maps. Each topic map relates
+ * topic UUIDs to arrays of partition IDs.
+ *
+ * Each entry in the array has a high bit which indicates that the broker is the leader
+ * for the given partition, as well as 31 low bits which contain the partition id. This
+ * works because partition IDs cannot be negative.
+ */
+public class BrokersToIsrs {
+ private final static int[] EMPTY = new int[0];
+
+ private final static int LEADER_FLAG = 0x8000_0000;
+
+ private final static int REPLICA_MASK = 0x7fff_ffff;
+
+ static class TopicPartition {
+ private final Uuid topicId;
+ private final int partitionId;
+
+ TopicPartition(Uuid topicId, int partitionId) {
+ this.topicId = topicId;
+ this.partitionId = partitionId;
+ }
+
+ public Uuid topicId() {
+ return topicId;
+ }
+
+ public int partitionId() {
+ return partitionId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof TopicPartition)) return false;
+ TopicPartition other = (TopicPartition) o;
+ return other.topicId.equals(topicId) && other.partitionId == partitionId;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(topicId, partitionId);
+ }
+
+ @Override
+ public String toString() {
+ return topicId + ":" + partitionId;
+ }
+ }
+
+ static class PartitionsOnReplicaIterator implements Iterator<TopicPartition> {
+ private final Iterator<Entry<Uuid, int[]>> iterator;
+ private final boolean leaderOnly;
+ private int offset = 0;
+ Uuid uuid = Uuid.ZERO_UUID;
+ int[] replicas = EMPTY;
+ private TopicPartition next = null;
+
+ PartitionsOnReplicaIterator(Map<Uuid, int[]> topicMap, boolean leaderOnly) {
+ this.iterator = topicMap.entrySet().iterator();
+ this.leaderOnly = leaderOnly;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) return true;
+ while (true) {
+ if (offset >= replicas.length) {
+ if (!iterator.hasNext()) return false;
+ offset = 0;
+ Entry<Uuid, int[]> entry = iterator.next();
+ uuid = entry.getKey();
+ replicas = entry.getValue();
+ }
+ int replica = replicas[offset++];
+ if ((!leaderOnly) || (replica & LEADER_FLAG) != 0) {
+ next = new TopicPartition(uuid, replica & REPLICA_MASK);
+ return true;
+ }
+ }
+ }
+
+ @Override
+ public TopicPartition next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ TopicPartition result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ * Partitions with no isr members appear in this map under id -1.
+ */
+ private final TimelineHashMap<Integer, TimelineHashMap<Uuid, int[]>> isrMembers;
+
+ BrokersToIsrs(SnapshotRegistry snapshotRegistry) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.isrMembers = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Update our records of a partition's ISR.
+ *
+ * @param topicId The topic ID of the partition.
+ * @param partitionId The partition ID of the partition.
+ * @param prevIsr The previous ISR, or null if the partition is new.
+ * @param nextIsr The new ISR, or null if the partition is being removed.
+ * @param prevLeader The previous leader, or -1 if the partition had no leader.
+ * @param nextLeader The new leader, or -1 if the partition now has no leader.
+ */
+ void update(Uuid topicId, int partitionId, int[] prevIsr, int[] nextIsr,
+ int prevLeader, int nextLeader) {
+ int[] prev;
+ if (prevIsr == null) {
+ prev = EMPTY;
+ } else {
+ if (prevLeader == -1) {
+ prev = Replicas.copyWith(prevIsr, -1);
+ } else {
+ prev = Replicas.clone(prevIsr);
+ }
+ Arrays.sort(prev);
+ }
+ int[] next;
+ if (nextIsr == null) {
+ next = EMPTY;
+ } else {
+ if (nextLeader == -1) {
+ next = Replicas.copyWith(nextIsr, -1);
+ } else {
+ next = Replicas.clone(nextIsr);
+ }
+ Arrays.sort(next);
+ }
+ int i = 0, j = 0;
+ while (true) {
+ if (i == prev.length) {
+ if (j == next.length) {
+ break;
+ }
+ int newReplica = next[j];
+ add(newReplica, topicId, partitionId, newReplica == nextLeader);
+ j++;
+ } else if (j == next.length) {
+ int prevReplica = prev[i];
+ remove(prevReplica, topicId, partitionId, prevReplica == prevLeader);
+ i++;
+ } else {
+ int prevReplica = prev[i];
+ int newReplica = next[j];
+ if (prevReplica < newReplica) {
+ remove(prevReplica, topicId, partitionId, prevReplica == prevLeader);
+ i++;
+ } else if (prevReplica > newReplica) {
+ add(newReplica, topicId, partitionId, newReplica == nextLeader);
+ j++;
+ } else {
+ boolean wasLeader = prevReplica == prevLeader;
+ boolean isLeader = prevReplica == nextLeader;
+ if (wasLeader != isLeader) {
+ change(prevReplica, topicId, partitionId, wasLeader, isLeader);
+ }
+ i++;
+ j++;
+ }
+ }
+ }
+ }
+
+ private void add(int brokerId, Uuid topicId, int newPartition, boolean leader) {
+ if (leader) {
+ newPartition = newPartition | LEADER_FLAG;
+ }
+ TimelineHashMap<Uuid, int[]> topicMap = isrMembers.get(brokerId);
+ if (topicMap == null) {
+ topicMap = new TimelineHashMap<>(snapshotRegistry, 0);
+ isrMembers.put(brokerId, topicMap);
+ }
+ int[] partitions = topicMap.get(topicId);
+ int[] newPartitions;
+ if (partitions == null) {
+ newPartitions = new int[1];
+ } else {
+ newPartitions = new int[partitions.length + 1];
+ System.arraycopy(partitions, 0, newPartitions, 0, partitions.length);
+ }
+ newPartitions[newPartitions.length - 1] = newPartition;
+ topicMap.put(topicId, newPartitions);
+ }
+
+ private void change(int brokerId, Uuid topicId, int partition,
+ boolean wasLeader, boolean isLeader) {
+ TimelineHashMap<Uuid, int[]> topicMap = isrMembers.get(brokerId);
+ if (topicMap == null) {
+ throw new RuntimeException("Broker " + brokerId + " has no isrMembers " +
+ "entry, so we can't change " + topicId + ":" + partition);
+ }
+ int[] partitions = topicMap.get(topicId);
+ if (partitions == null) {
+ throw new RuntimeException("Broker " + brokerId + " has no " +
+ "entry in isrMembers for topic " + topicId);
+ }
+ int[] newPartitions = new int[partitions.length];
+ int target = wasLeader ? partition | LEADER_FLAG : partition;
+ for (int i = 0; i < partitions.length; i++) {
+ int cur = partitions[i];
+ if (cur == target) {
+ newPartitions[i] = isLeader ? partition | LEADER_FLAG : partition;
Review comment:
since this is stored in a `TimelineHashMap` it must be treated as immutable. We can't modify the past.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578817527
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef.ConfigKey;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource.Type;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.APPEND;
+
+public class ConfigurationControlManager {
+ private final Logger log;
+ private final SnapshotRegistry snapshotRegistry;
+ private final Map<ConfigResource.Type, ConfigDef> configDefs;
+ private final TimelineHashMap<ConfigResource, TimelineHashMap<String, String>> configData;
+
+ ConfigurationControlManager(LogContext logContext,
+ SnapshotRegistry snapshotRegistry,
+ Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.log = logContext.logger(ConfigurationControlManager.class);
+ this.snapshotRegistry = snapshotRegistry;
+ this.configDefs = configDefs;
+ this.configData = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Determine the result of applying a batch of incremental configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * Note that there can only be one result per ConfigResource. So if you try to modify
+ * several keys and one modification fails, the whole ConfigKey fails and nothing gets
+ * changed.
+ *
+ * @param configChanges Maps each resource to a map from config keys to
+ * operation data.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, Entry<OpType, String>>> resourceEntry :
+ configChanges.entrySet()) {
+ incrementalAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void incrementalAlterConfigResource(ConfigResource configResource,
+ Map<String, Entry<OpType, String>> keysToOps,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ for (Entry<String, Entry<OpType, String>> keysToOpsEntry : keysToOps.entrySet()) {
+ String key = keysToOpsEntry.getKey();
+ String currentValue = null;
+ TimelineHashMap<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs != null) {
+ currentValue = currentConfigs.get(key);
+ }
+ String newValue = currentValue;
+ Entry<OpType, String> opTypeAndNewValue = keysToOpsEntry.getValue();
+ OpType opType = opTypeAndNewValue.getKey();
+ String opValue = opTypeAndNewValue.getValue();
+ switch (opType) {
+ case SET:
+ newValue = opValue;
+ break;
+ case DELETE:
+ if (opValue != null) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_REQUEST, "A DELETE op was given with a " +
+ "non-null value."));
+ return;
+ }
+ newValue = null;
+ break;
+ case APPEND:
+ case SUBTRACT:
+ if (!isSplittable(configResource.type(), key)) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_CONFIG, "Can't " + opType + " to " +
+ "key " + key + " because its type is not LIST."));
+ return;
+ }
+ List<String> newValueParts = getParts(newValue, key, configResource);
+ if (opType == APPEND) {
+ if (!newValueParts.contains(opValue)) {
+ newValueParts.add(opValue);
+ }
+ newValue = String.join(",", newValueParts);
+ } else if (newValueParts.remove(opValue)) {
+ newValue = String.join(",", newValueParts);
+ }
+ break;
+ }
+ if (!Objects.equals(currentValue, newValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ /**
+ * Determine the result of applying a batch of legacy configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * @param newConfigs The new configurations to install for each resource.
+ * All existing configurations will be overwritten.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, String>> resourceEntry :
+ newConfigs.entrySet()) {
+ legacyAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void legacyAlterConfigResource(ConfigResource configResource,
+ Map<String, String> newConfigs,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ Map<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs == null) {
+ currentConfigs = Collections.emptyMap();
+ }
+ for (Entry<String, String> entry : newConfigs.entrySet()) {
+ String key = entry.getKey();
+ String newValue = entry.getValue();
+ String currentValue = currentConfigs.get(key);
+ if (!Objects.equals(newValue, currentValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ for (String key : currentConfigs.keySet()) {
+ if (!newConfigs.containsKey(key)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(null), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ private List<String> getParts(String value, String key, ConfigResource configResource) {
+ if (value == null) {
+ value = getConfigValueDefault(configResource.type(), key);
+ }
+ List<String> parts = new ArrayList<>();
+ if (value == null) {
+ return parts;
+ }
+ String[] splitValues = value.split(",");
+ for (String splitValue : splitValues) {
+ if (!splitValue.isEmpty()) {
+ parts.add(splitValue);
+ }
+ }
+ return parts;
+ }
+
+ static ApiError checkConfigResource(ConfigResource configResource) {
+ switch (configResource.type()) {
+ case BROKER_LOGGER:
+ // We do not handle resources of type BROKER_LOGGER in
+ // ConfigurationControlManager, since they are not persisted to the
+ // metadata log.
+ //
+ // When using incrementalAlterConfigs, we handle changes to BROKER_LOGGER
+ // in ControllerApis.scala. When using the legacy alterConfigs,
+ // BROKER_LOGGER is not supported at all.
+ return new ApiError(Errors.INVALID_REQUEST, "Unsupported " +
+ "configuration resource type BROKER_LOGGER ");
+ case BROKER:
+ if (!configResource.name().isEmpty()) {
Review comment:
A Resource with type BROKER and an empty name represents a cluster configuration that applies to all brokers. I'll add a comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577895640
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
+ movingLeadersForShutDown = !records.isEmpty();
+ }
+ ControllerResult<BrokerHeartbeatReply> result = clusterControl.
+ processBrokerHeartbeat(request, lastCommittedOffset,
+ movingLeadersForShutDown);
+ if (movingLeadersForShutDown && !result.response().shouldShutDown()) {
+ updateShutdownOffset = true;
+ }
+ if (result.response().isFenced()) {
+ replicationControl.removeFromIsr(request.brokerId(), records);
Review comment:
Hmm... We only fence the broker once controlled shutdown has completed. If we fenced it immediately that would be disruptive to clients since they wouldn't be able to continue fetching from it until the leaderships have moved. Basically immediate fencing is the NON-controlled shutdown path....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577278241
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
+ movingLeadersForShutDown = !records.isEmpty();
+ }
+ ControllerResult<BrokerHeartbeatReply> result = clusterControl.
+ processBrokerHeartbeat(request, lastCommittedOffset,
+ movingLeadersForShutDown);
+ if (movingLeadersForShutDown && !result.response().shouldShutDown()) {
+ updateShutdownOffset = true;
+ }
+ if (result.response().isFenced()) {
+ replicationControl.removeFromIsr(request.brokerId(), records);
Review comment:
Well, in ZK based approach, in response to a controlled shutdown, the controller changes ISR and the leader. Here, it seems that `result.response().isFenced()` is not always true if the broker heartbeat indicates the intention to shut down?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577214559
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
+ isFenced = true;
+ shouldShutdown = true;
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ } else {
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ }
+ }
+ heartbeatManager.touch(brokerId, isFenced, request.currentMetadataOffset());
+ ControllerResult<BrokerHeartbeatReply> result = new ControllerResult<>(records,
+ new BrokerHeartbeatReply(isCaughtUp, isFenced, shouldShutdown));
+ if (log.isTraceEnabled()) {
+ log.trace("processed broker heartbeat {}. lastCommittedOffset = {}, " +
+ "movingLeadersForShutDown = {}, result = {}", request,
+ lastCommittedOffset, movingLeadersForShutDown, result);
+ }
+ return result;
+ }
+
+ public void replay(RegisterBrokerRecord record) {
+ int brokerId = record.brokerId();
+ List<Endpoint> listeners = new ArrayList<>();
+ for (RegisterBrokerRecord.BrokerEndpoint endpoint : record.endPoints()) {
+ listeners.add(new Endpoint(endpoint.name(),
+ SecurityProtocol.forId(endpoint.securityProtocol()),
+ endpoint.host(), endpoint.port()));
+ }
+ Map<String, VersionRange> features = new HashMap<>();
+ for (RegisterBrokerRecord.BrokerFeature feature : record.features()) {
+ features.put(feature.name(), new VersionRange(
+ feature.minSupportedVersion(), feature.maxSupportedVersion()));
+ }
+ // Normally, all newly registered brokers start off in the fenced state.
+ // If this registration record is for a broker incarnation that was already
+ // registered, though, we preserve the existing fencing state.
+ boolean fenced = true;
+ BrokerRegistration prevRegistration = brokerRegistrations.get(brokerId);
+ if (prevRegistration != null &&
+ prevRegistration.incarnationId().equals(record.incarnationId())) {
+ fenced = prevRegistration.fenced();
+ }
+ // Update broker registrations.
+ brokerRegistrations.put(brokerId, new BrokerRegistration(brokerId,
+ record.brokerEpoch(), record.incarnationId(), listeners, features,
+ Optional.ofNullable(record.rack()), fenced));
+
+ if (prevRegistration == null) {
+ log.info("Registered new broker: {}", record);
+ } else if (prevRegistration.incarnationId().equals(record.incarnationId())) {
+ log.info("Re-registered broker incarnation: {}", record);
+ } else {
+ log.info("Re-registered broker id {}: {}", brokerId, record);
+ }
+ }
+
+ public void replay(UnregisterBrokerRecord record) {
+ int brokerId = record.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration found for that id", record.toString()));
+ } else if (registration.epoch() != record.brokerEpoch()) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration with that epoch found", record.toString()));
+ } else {
+ brokerRegistrations.remove(brokerId);
+ log.info("Unregistered broker: {}", record);
+ }
+ }
+
+ public void replay(FenceBrokerRecord record) {
+ int brokerId = record.id();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration found for that id", record.toString()));
+ } else if (registration.epoch() != record.epoch()) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration with that epoch found", record.toString()));
+ } else {
+ brokerRegistrations.put(brokerId, registration.cloneWithFencing(true));
+ log.info("Fenced broker: {}", record);
+ }
+ }
+
+ public void replay(UnfenceBrokerRecord record) {
+ int brokerId = record.id();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration found for that id", record.toString()));
+ } else if (registration.epoch() != record.epoch()) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration with that epoch found", record.toString()));
+ } else {
+ brokerRegistrations.put(brokerId, registration.cloneWithFencing(false));
Review comment:
Good point. The appropriate place to handle this will be in the broker heartbeat handling code, since that is where the active controller unfences brokers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577125115
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
Review comment:
This code is only executed by the followers. It is true that the leader already applied these log messages but these nodes are not the leader.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578065045
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
Review comment:
Are all records generated in a single ControllerWriteEvent appended to the metadata log atomically?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577263866
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
I thought the issue was that if changing leaderEpoch to Long requires a log format change for user data, it has significant performance impact such as down conversion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579522827
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
Review comment:
The leader epoch is managed implicitly -- every time a PartitionChangeRecord appears, the epoch is bumped if the leader is not NO_LEADER_CHANGE.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578648887
##########
File path: metadata/src/main/java/org/apache/kafka/metadata/BrokerHeartbeatReply.java
##########
@@ -31,16 +31,23 @@
*/
private final boolean isFenced;
+ /**
+ * True if the heartbeat reply should tell the broker that it should shut down.
Review comment:
fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573236986
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
Review comment:
Thanks... that is a good catch. Yes, it's a bit more efficient if the statements are nested. I will refactor this out into a separate function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577136289
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
It is an int and will be in 2.8, but I think that's a mistake (as discussed in the mailing list) and we should plan to make this 64 bit in the near future to avoid overflow issues.
So the controller code treats it as a long... cc @hachikuji
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578003900
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
It seems that using long for leaderEpoch in the log requires a bigger discussion. Could we use int for now?
colin:
For some reason, github is not giving me a reply button on your comment. So I will add my reply here.
I do think we should harmonize this, but I think it would be better to do that when we get rid of MetaLogShim. We've had a plan to get rid of the shim layer for a while but we just didn't have time to do it this week. So let's plan to do it then, if that makes sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577922334
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
Review comment:
the standby controller must replay all the committed records before becoming active. so, there is no unfinished work to be done at this point.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577262652
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
Review comment:
It seems that we are logging at the debug level. I am wondering if we should log at WARN as before in ZK based appoach.
```
if (exception instanceof ApiException) {
log.debug("{}: failed with {} in {} us", name,
exception.getClass().getSimpleName(), deltaUs);
return exception;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579564268
##########
File path: metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.kafka.common.metadata.FeatureLevelRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.FeatureMap;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.apache.kafka.timeline.TimelineHashSet;
+
+
+public class FeatureControlManager {
+ /**
+ * The features supported by this controller's software.
+ */
+ private final Map<String, VersionRange> supportedFeatures;
+
+ /**
+ * Maps feature names to finalized version ranges.
+ */
+ private final TimelineHashMap<String, VersionRange> finalizedVersions;
+
+ /**
+ * The latest feature epoch.
+ */
+ private final TimelineHashSet<Long> epoch;
+
+ FeatureControlManager(Map<String, VersionRange> supportedFeatures,
+ SnapshotRegistry snapshotRegistry) {
+ this.supportedFeatures = supportedFeatures;
+ this.finalizedVersions = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.epoch = new TimelineHashSet<>(snapshotRegistry, 0);
+ }
+
+ ControllerResult<Map<String, ApiError>> updateFeatures(
+ Map<String, VersionRange> updates, Set<String> downgradeables,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures) {
+ TreeMap<String, ApiError> results = new TreeMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (Entry<String, VersionRange> entry : updates.entrySet()) {
+ results.put(entry.getKey(), updateFeature(entry.getKey(), entry.getValue(),
+ downgradeables.contains(entry.getKey()), brokerFeatures, records));
+ }
+ return new ControllerResult<>(records, results);
+ }
+
+ private ApiError updateFeature(String featureName,
+ VersionRange newRange,
+ boolean downgradeable,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures,
+ List<ApiMessageAndVersion> records) {
+ if (newRange.min() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The lower value for the new range cannot be less than 1.");
+ }
+ if (newRange.max() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The upper value for the new range cannot be less than 1.");
+ }
+ VersionRange localRange = supportedFeatures.get(featureName);
Review comment:
let's revisit this after 2.8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579564579
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
Review comment:
RIght now the answer is yes. Eventually we plan on supporting multiple batches.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578603931
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.Gauge;
+import com.yammer.metrics.core.Histogram;
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.MetricsRegistry;
+
+
+public final class QuorumControllerMetrics implements ControllerMetrics {
+ private final static MetricName ACTIVE_CONTROLLER_COUNT = new MetricName(
+ "kafka.controller", "KafkaController", "ActiveControllerCount", null);
+ private final static MetricName EVENT_QUEUE_TIME_MS = new MetricName(
+ "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
Review comment:
Good point. I will fix it so that EventQueueTimeMs has its original meaning.
For now, I have added a metric called EventQueueProcessingTimeMs which deals with processing time. I do want to do the per-state tracking but I don't think we have time right now
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577894071
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
Review comment:
Note: this has been changed to unregisterBroker as per the mailing list discussion.
It's OK to have a future that returns void. That just means you either get success, or an error (there is no other result). Which is consistent with UnregisterBrokerResponse.json, etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577167409
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
Review comment:
good point, will fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579438984
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
Review comment:
Hmm, merge bumps up the leaderEpoch. It seems that this needs to be persisted in the metadata log?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577268593
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
Review comment:
Sorry, I meant `decommissionBroker`. It seems that decommissionBroker request should we a response too, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579374802
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,875 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ ElectLeadersResponseData response = new ElectLeadersResponseData();
+ for (TopicPartitions topic : request.topicPartitions()) {
+ ReplicaElectionResult topicResults =
+ new ReplicaElectionResult().setTopic(topic.topic());
+ response.replicaElectionResults().add(topicResults);
+ for (int partitionId : topic.partitions()) {
+ ApiError error = electLeader(topic.topic(), partitionId, unclean, records);
+ topicResults.partitionResult().add(new PartitionResult().
+ setPartitionId(partitionId).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ static boolean electionIsUnclean(byte electionType) {
+ ElectionType type;
+ try {
+ type = ElectionType.valueOf(electionType);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidRequestException("Unknown election type " + (int) electionType);
+ }
+ return type == ElectionType.UNCLEAN;
+ }
+
+ ApiError electLeader(String topic, int partitionId, boolean unclean,
+ List<ApiMessageAndVersion> records) {
+ Uuid topicId = topicsByName.get(topic);
+ if (topicId == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic as " + topic);
+ }
+ TopicControlInfo topicInfo = topics.get(topicId);
+ if (topicInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic id as " + topicId);
+ }
+ PartitionControlInfo partitionInfo = topicInfo.parts.get(partitionId);
+ if (partitionInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such partition as " + topic + "-" + partitionId);
+ }
+ int newLeader = chooseNewLeader(partitionInfo, partitionInfo.isr, unclean);
+ if (newLeader == partitionInfo.leader) {
+ if (newLeader < 0) {
+ return new ApiError(Errors.LEADER_NOT_AVAILABLE,
+ "Unable to find any leader for the partition.");
+ } else {
+ return ApiError.NONE;
+ }
+ } else {
+ int[] newIsr = partitionInfo.isr;
+ if (!Replicas.contains(partitionInfo.isr, newLeader)) {
+ newIsr = new int[] {newLeader};
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionId).
+ setTopicId(topicId).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partitionInfo.leaderEpoch + 1).
+ setPartitionEpoch(partitionInfo.partitionEpoch + 1), (short) 0));
+ return ApiError.NONE;
+ }
+ }
+
+ int chooseNewLeader(PartitionControlInfo partition, int[] newIsr, boolean unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (clusterControl.unfenced(replica)) {
+ return replica;
+ }
+ }
+ }
+ return -1;
+ }
+
+ ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request, long lastCommittedOffset) {
+ int brokerId = request.brokerId();
+ long brokerEpoch = request.brokerEpoch();
+ clusterControl.checkBrokerEpoch(brokerId, brokerEpoch);
+ BrokerHeartbeatManager heartbeatManager = clusterControl.heartbeatManager();
+ BrokerControlStates states = heartbeatManager.calculateNextBrokerState(brokerId,
+ request, lastCommittedOffset, () -> brokersToIsrs.hasLeaderships(brokerId));
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ if (states.current() != states.next()) {
+ switch (states.next()) {
+ case FENCED:
+ handleBrokerFenced(brokerId, records);
+ break;
+ case UNFENCED:
+ handleBrokerUnfenced(brokerId, brokerEpoch, records);
+ break;
+ case CONTROLLED_SHUTDOWN:
+ handleNodeDeactivated(brokerId, records);
Review comment:
This is resolved in the latest version of the code, where we disable metadata updates on the shutting down broker before starting controlled shutdown, and bump the leader epoch of all partitions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579565540
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
Review comment:
ack. I fixed this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577918400
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,519 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.shutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ /**
+ * Move a broker into shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param deferred If this is true, the broker can shut down immediately.
+ * Otherwise, the broker must wait for some replicas to be moved.
+ */
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutDownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ // If the broker is fenced, we don't wait for controlled shutdown, since
+ // its leaders have already been moved away.
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
+ if (!broker.shuttingDown()) {
+ log.trace("broker {} should not shut down because it is not in the " +
+ "shutdown state.", brokerId);
+ return false;
+ }
+ long lowestActiveOffset = lowestActiveOffset();
+ if (lowestActiveOffset >= broker.shutDownOffset) {
+ log.debug("broker {} should shut down. The current lowest active offset, {} " +
+ "is greater or equal to the broker shutdown offset of {}", brokerId,
+ lowestActiveOffset, broker.shutDownOffset);
+ return true;
+ } else {
+ log.debug("broker {} must wait to shut down. The current lowest active offset, {} " +
+ "is lower than the broker shutdown offset of {}", brokerId,
+ lowestActiveOffset, broker.shutDownOffset);
+ return false;
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Update the offset at which a broker should shut down.
+ *
+ * The controller invokes this function after it has scheduled a batch of records to
+ * be written to move leaders off of the broker.
+ *
+ * @param brokerId The broker id.
+ * @param offset The new offset.
+ */
+ void updateShutdownOffset(int brokerId, long offset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ active.remove(broker);
+ broker.shutDownOffset = offset;
+ log.debug("Updated the shutdown offset for broker {} to {}.", brokerId, offset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Fenced the broker which was least recently contacted, if needed.
+ *
+ * @return -1 if no broker was fenced; the broker ID otherwise. If the return
+ * value is non-negative, this function should be called again.
+ */
+ int maybeFenceLeastRecentlyContacted() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null || hasValidSession(broker)) {
+ return -1;
+ } else {
+ unfenced.remove(broker);
+ return broker.id();
+ }
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
Review comment:
good point. Since we don't have much time for 2.8, I will add a TODO for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573235373
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
Review comment:
This doesn't need to be public because it's only accessed from within the controller package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577971959
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,916 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import static java.util.concurrent.TimeUnit.MICROSECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = NANOSECONDS.convert(18, TimeUnit.SECONDS);
+ private ControllerMetrics controllerMetrics = null;
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public Builder setMetrics(ControllerMetrics controllerMetrics) {
+ this.controllerMetrics = controllerMetrics;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ if (controllerMetrics == null) {
+ controllerMetrics = (ControllerMetrics) Class.forName(
+ "org.apache.kafka.controller.MockControllerMetrics").getConstructor().newInstance();
Review comment:
Maybe it would be better to check for null and exit out if it is unset -- otherwise we see this, which is not ideal:
```
Fatal error during controller startup. Prepare to shutdown (kafka.server.ControllerServer)
java.lang.ClassNotFoundException: org.apache.kafka.controller.MockControllerMetrics
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578850804
##########
File path: metadata/src/test/java/org/apache/kafka/controller/MockControllerMetrics.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+
+public final class MockControllerMetrics implements ControllerMetrics {
Review comment:
It's used in unit tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578693886
##########
File path: metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.kafka.common.metadata.FeatureLevelRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.FeatureMap;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.apache.kafka.timeline.TimelineHashSet;
+
+
+public class FeatureControlManager {
+ /**
+ * The features supported by this controller's software.
+ */
+ private final Map<String, VersionRange> supportedFeatures;
+
+ /**
+ * Maps feature names to finalized version ranges.
+ */
+ private final TimelineHashMap<String, VersionRange> finalizedVersions;
+
+ /**
+ * The latest feature epoch.
+ */
+ private final TimelineHashSet<Long> epoch;
+
+ FeatureControlManager(Map<String, VersionRange> supportedFeatures,
+ SnapshotRegistry snapshotRegistry) {
+ this.supportedFeatures = supportedFeatures;
+ this.finalizedVersions = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.epoch = new TimelineHashSet<>(snapshotRegistry, 0);
+ }
+
+ ControllerResult<Map<String, ApiError>> updateFeatures(
+ Map<String, VersionRange> updates, Set<String> downgradeables,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures) {
+ TreeMap<String, ApiError> results = new TreeMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (Entry<String, VersionRange> entry : updates.entrySet()) {
+ results.put(entry.getKey(), updateFeature(entry.getKey(), entry.getValue(),
+ downgradeables.contains(entry.getKey()), brokerFeatures, records));
+ }
+ return new ControllerResult<>(records, results);
+ }
+
+ private ApiError updateFeature(String featureName,
+ VersionRange newRange,
+ boolean downgradeable,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures,
+ List<ApiMessageAndVersion> records) {
+ if (newRange.min() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The lower value for the new range cannot be less than 1.");
+ }
+ if (newRange.max() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The upper value for the new range cannot be less than 1.");
+ }
+ VersionRange localRange = supportedFeatures.get(featureName);
Review comment:
This can be revisited later. When finalizing a feature, should be consider other controller's supported features too?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,925 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import static java.util.concurrent.TimeUnit.MICROSECONDS;
+import static java.util.concurrent.TimeUnit.NANOSECONDS;
+
+
+public final class QuorumController implements Controller {
Review comment:
Could we add some comments for this class?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,875 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ ElectLeadersResponseData response = new ElectLeadersResponseData();
+ for (TopicPartitions topic : request.topicPartitions()) {
+ ReplicaElectionResult topicResults =
+ new ReplicaElectionResult().setTopic(topic.topic());
+ response.replicaElectionResults().add(topicResults);
+ for (int partitionId : topic.partitions()) {
+ ApiError error = electLeader(topic.topic(), partitionId, unclean, records);
+ topicResults.partitionResult().add(new PartitionResult().
+ setPartitionId(partitionId).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ static boolean electionIsUnclean(byte electionType) {
+ ElectionType type;
+ try {
+ type = ElectionType.valueOf(electionType);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidRequestException("Unknown election type " + (int) electionType);
+ }
+ return type == ElectionType.UNCLEAN;
+ }
+
+ ApiError electLeader(String topic, int partitionId, boolean unclean,
+ List<ApiMessageAndVersion> records) {
+ Uuid topicId = topicsByName.get(topic);
+ if (topicId == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic as " + topic);
+ }
+ TopicControlInfo topicInfo = topics.get(topicId);
+ if (topicInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic id as " + topicId);
+ }
+ PartitionControlInfo partitionInfo = topicInfo.parts.get(partitionId);
+ if (partitionInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such partition as " + topic + "-" + partitionId);
+ }
+ int newLeader = chooseNewLeader(partitionInfo, partitionInfo.isr, unclean);
+ if (newLeader == partitionInfo.leader) {
+ if (newLeader < 0) {
+ return new ApiError(Errors.LEADER_NOT_AVAILABLE,
+ "Unable to find any leader for the partition.");
+ } else {
+ return ApiError.NONE;
+ }
+ } else {
+ int[] newIsr = partitionInfo.isr;
+ if (!Replicas.contains(partitionInfo.isr, newLeader)) {
+ newIsr = new int[] {newLeader};
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionId).
+ setTopicId(topicId).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partitionInfo.leaderEpoch + 1).
+ setPartitionEpoch(partitionInfo.partitionEpoch + 1), (short) 0));
+ return ApiError.NONE;
+ }
+ }
+
+ int chooseNewLeader(PartitionControlInfo partition, int[] newIsr, boolean unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (clusterControl.unfenced(replica)) {
+ return replica;
+ }
+ }
+ }
+ return -1;
+ }
+
+ ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request, long lastCommittedOffset) {
+ int brokerId = request.brokerId();
+ long brokerEpoch = request.brokerEpoch();
+ clusterControl.checkBrokerEpoch(brokerId, brokerEpoch);
+ BrokerHeartbeatManager heartbeatManager = clusterControl.heartbeatManager();
+ BrokerControlStates states = heartbeatManager.calculateNextBrokerState(brokerId,
+ request, lastCommittedOffset, () -> brokersToIsrs.hasLeaderships(brokerId));
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ if (states.current() != states.next()) {
+ switch (states.next()) {
+ case FENCED:
+ handleBrokerFenced(brokerId, records);
+ break;
+ case UNFENCED:
+ handleBrokerUnfenced(brokerId, brokerEpoch, records);
+ break;
+ case CONTROLLED_SHUTDOWN:
+ handleNodeDeactivated(brokerId, records);
Review comment:
> It seems like the remaining behavioral difference is that the new code will, if no other leader can be chosen, set the leader to -1 (offline). If we don't do this, controlled shutdown easily gets stuck if there are any partitions with replication factor = 1. Maybe we can tune this a bit later?
It's fine to revisit that later. The tradeoff is that if we wait, it slightly increases the probability of availability since another replica could join isr.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
Review comment:
I think we need to handle preferred leader election in a special way. For example, if the assigned replicas are 1,2,3, isr is 2,3 and the current leader is 3, when doing preferred leader election, we want to keep the leader as 3 instead of changing it to 2.
##########
File path: metadata/src/test/java/org/apache/kafka/controller/MockControllerMetrics.java
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+
+public final class MockControllerMetrics implements ControllerMetrics {
Review comment:
This class seems never used?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef.ConfigKey;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource.Type;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.APPEND;
+
+public class ConfigurationControlManager {
+ private final Logger log;
+ private final SnapshotRegistry snapshotRegistry;
+ private final Map<ConfigResource.Type, ConfigDef> configDefs;
+ private final TimelineHashMap<ConfigResource, TimelineHashMap<String, String>> configData;
+
+ ConfigurationControlManager(LogContext logContext,
+ SnapshotRegistry snapshotRegistry,
+ Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.log = logContext.logger(ConfigurationControlManager.class);
+ this.snapshotRegistry = snapshotRegistry;
+ this.configDefs = configDefs;
+ this.configData = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Determine the result of applying a batch of incremental configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * Note that there can only be one result per ConfigResource. So if you try to modify
+ * several keys and one modification fails, the whole ConfigKey fails and nothing gets
+ * changed.
+ *
+ * @param configChanges Maps each resource to a map from config keys to
+ * operation data.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, Entry<OpType, String>>> resourceEntry :
+ configChanges.entrySet()) {
+ incrementalAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void incrementalAlterConfigResource(ConfigResource configResource,
+ Map<String, Entry<OpType, String>> keysToOps,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ for (Entry<String, Entry<OpType, String>> keysToOpsEntry : keysToOps.entrySet()) {
+ String key = keysToOpsEntry.getKey();
+ String currentValue = null;
+ TimelineHashMap<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs != null) {
+ currentValue = currentConfigs.get(key);
+ }
+ String newValue = currentValue;
+ Entry<OpType, String> opTypeAndNewValue = keysToOpsEntry.getValue();
+ OpType opType = opTypeAndNewValue.getKey();
+ String opValue = opTypeAndNewValue.getValue();
+ switch (opType) {
+ case SET:
+ newValue = opValue;
+ break;
+ case DELETE:
+ if (opValue != null) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_REQUEST, "A DELETE op was given with a " +
+ "non-null value."));
+ return;
+ }
+ newValue = null;
+ break;
+ case APPEND:
+ case SUBTRACT:
+ if (!isSplittable(configResource.type(), key)) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_CONFIG, "Can't " + opType + " to " +
+ "key " + key + " because its type is not LIST."));
+ return;
+ }
+ List<String> newValueParts = getParts(newValue, key, configResource);
+ if (opType == APPEND) {
+ if (!newValueParts.contains(opValue)) {
+ newValueParts.add(opValue);
+ }
+ newValue = String.join(",", newValueParts);
+ } else if (newValueParts.remove(opValue)) {
+ newValue = String.join(",", newValueParts);
+ }
+ break;
+ }
+ if (!Objects.equals(currentValue, newValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ /**
+ * Determine the result of applying a batch of legacy configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * @param newConfigs The new configurations to install for each resource.
+ * All existing configurations will be overwritten.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, String>> resourceEntry :
+ newConfigs.entrySet()) {
+ legacyAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void legacyAlterConfigResource(ConfigResource configResource,
+ Map<String, String> newConfigs,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ Map<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs == null) {
+ currentConfigs = Collections.emptyMap();
+ }
+ for (Entry<String, String> entry : newConfigs.entrySet()) {
+ String key = entry.getKey();
+ String newValue = entry.getValue();
+ String currentValue = currentConfigs.get(key);
+ if (!Objects.equals(newValue, currentValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ for (String key : currentConfigs.keySet()) {
+ if (!newConfigs.containsKey(key)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(null), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ private List<String> getParts(String value, String key, ConfigResource configResource) {
+ if (value == null) {
+ value = getConfigValueDefault(configResource.type(), key);
+ }
+ List<String> parts = new ArrayList<>();
+ if (value == null) {
+ return parts;
+ }
+ String[] splitValues = value.split(",");
+ for (String splitValue : splitValues) {
+ if (!splitValue.isEmpty()) {
+ parts.add(splitValue);
+ }
+ }
+ return parts;
+ }
+
+ static ApiError checkConfigResource(ConfigResource configResource) {
+ switch (configResource.type()) {
+ case BROKER_LOGGER:
+ // We do not handle resources of type BROKER_LOGGER in
+ // ConfigurationControlManager, since they are not persisted to the
+ // metadata log.
+ //
+ // When using incrementalAlterConfigs, we handle changes to BROKER_LOGGER
+ // in ControllerApis.scala. When using the legacy alterConfigs,
+ // BROKER_LOGGER is not supported at all.
+ return new ApiError(Errors.INVALID_REQUEST, "Unsupported " +
+ "configuration resource type BROKER_LOGGER ");
+ case BROKER:
+ if (!configResource.name().isEmpty()) {
+ try {
+ int brokerId = Integer.parseInt(configResource.name());
+ if (brokerId < 0) {
+ return new ApiError(Errors.INVALID_REQUEST, "Illegal " +
+ "negative broker ID in BROKER resource.");
+ }
+ } catch (NumberFormatException e) {
+ return new ApiError(Errors.INVALID_REQUEST, "Illegal " +
+ "non-integral BROKER resource type name.");
+ }
+ }
+ return ApiError.NONE;
+ case TOPIC:
+ if (!configResource.name().isEmpty()) {
Review comment:
An empty topic name currently results in an INVALID_REQUEST error.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef.ConfigKey;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource.Type;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.APPEND;
+
+public class ConfigurationControlManager {
+ private final Logger log;
+ private final SnapshotRegistry snapshotRegistry;
+ private final Map<ConfigResource.Type, ConfigDef> configDefs;
+ private final TimelineHashMap<ConfigResource, TimelineHashMap<String, String>> configData;
+
+ ConfigurationControlManager(LogContext logContext,
+ SnapshotRegistry snapshotRegistry,
+ Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.log = logContext.logger(ConfigurationControlManager.class);
+ this.snapshotRegistry = snapshotRegistry;
+ this.configDefs = configDefs;
+ this.configData = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Determine the result of applying a batch of incremental configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * Note that there can only be one result per ConfigResource. So if you try to modify
+ * several keys and one modification fails, the whole ConfigKey fails and nothing gets
+ * changed.
+ *
+ * @param configChanges Maps each resource to a map from config keys to
+ * operation data.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, Entry<OpType, String>>> resourceEntry :
+ configChanges.entrySet()) {
+ incrementalAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void incrementalAlterConfigResource(ConfigResource configResource,
+ Map<String, Entry<OpType, String>> keysToOps,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ for (Entry<String, Entry<OpType, String>> keysToOpsEntry : keysToOps.entrySet()) {
+ String key = keysToOpsEntry.getKey();
+ String currentValue = null;
+ TimelineHashMap<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs != null) {
+ currentValue = currentConfigs.get(key);
+ }
+ String newValue = currentValue;
+ Entry<OpType, String> opTypeAndNewValue = keysToOpsEntry.getValue();
+ OpType opType = opTypeAndNewValue.getKey();
+ String opValue = opTypeAndNewValue.getValue();
+ switch (opType) {
+ case SET:
+ newValue = opValue;
+ break;
+ case DELETE:
+ if (opValue != null) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_REQUEST, "A DELETE op was given with a " +
+ "non-null value."));
+ return;
+ }
+ newValue = null;
+ break;
+ case APPEND:
+ case SUBTRACT:
+ if (!isSplittable(configResource.type(), key)) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_CONFIG, "Can't " + opType + " to " +
+ "key " + key + " because its type is not LIST."));
+ return;
+ }
+ List<String> newValueParts = getParts(newValue, key, configResource);
+ if (opType == APPEND) {
+ if (!newValueParts.contains(opValue)) {
+ newValueParts.add(opValue);
+ }
+ newValue = String.join(",", newValueParts);
+ } else if (newValueParts.remove(opValue)) {
+ newValue = String.join(",", newValueParts);
+ }
+ break;
+ }
+ if (!Objects.equals(currentValue, newValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ /**
+ * Determine the result of applying a batch of legacy configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * @param newConfigs The new configurations to install for each resource.
+ * All existing configurations will be overwritten.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, String>> resourceEntry :
+ newConfigs.entrySet()) {
+ legacyAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void legacyAlterConfigResource(ConfigResource configResource,
+ Map<String, String> newConfigs,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ Map<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs == null) {
+ currentConfigs = Collections.emptyMap();
+ }
+ for (Entry<String, String> entry : newConfigs.entrySet()) {
+ String key = entry.getKey();
+ String newValue = entry.getValue();
+ String currentValue = currentConfigs.get(key);
+ if (!Objects.equals(newValue, currentValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ for (String key : currentConfigs.keySet()) {
+ if (!newConfigs.containsKey(key)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(null), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ private List<String> getParts(String value, String key, ConfigResource configResource) {
+ if (value == null) {
+ value = getConfigValueDefault(configResource.type(), key);
+ }
+ List<String> parts = new ArrayList<>();
+ if (value == null) {
+ return parts;
+ }
+ String[] splitValues = value.split(",");
+ for (String splitValue : splitValues) {
+ if (!splitValue.isEmpty()) {
+ parts.add(splitValue);
+ }
+ }
+ return parts;
+ }
+
+ static ApiError checkConfigResource(ConfigResource configResource) {
+ switch (configResource.type()) {
+ case BROKER_LOGGER:
+ // We do not handle resources of type BROKER_LOGGER in
+ // ConfigurationControlManager, since they are not persisted to the
+ // metadata log.
+ //
+ // When using incrementalAlterConfigs, we handle changes to BROKER_LOGGER
+ // in ControllerApis.scala. When using the legacy alterConfigs,
+ // BROKER_LOGGER is not supported at all.
+ return new ApiError(Errors.INVALID_REQUEST, "Unsupported " +
+ "configuration resource type BROKER_LOGGER ");
+ case BROKER:
+ if (!configResource.name().isEmpty()) {
Review comment:
An empty broker currently results in an INVALID_REQUEST error.
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTestEnv.java
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.metalog.LocalLogManagerTestEnv;
+import org.apache.kafka.test.TestUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+
+public class QuorumControllerTestEnv implements AutoCloseable {
+ private static final Logger log =
+ LoggerFactory.getLogger(QuorumControllerTestEnv.class);
+
+ private final List<QuorumController> controllers;
+
+ public QuorumControllerTestEnv(LocalLogManagerTestEnv logEnv,
+ Consumer<QuorumController.Builder> builderConsumer)
+ throws Exception {
+ int numControllers = logEnv.logManagers().size();
+ this.controllers = new ArrayList<>(numControllers);
+ try {
+ for (int i = 0; i < numControllers; i++) {
+ QuorumController.Builder builder = new QuorumController.Builder(i);
+ builder.setLogManager(logEnv.logManagers().get(i));
+ builderConsumer.accept(builder);
+ this.controllers.add(builder.build());
+ }
+ } catch (Exception e) {
+ close();
+ throw e;
+ }
+ }
+
+ QuorumController activeController() throws InterruptedException {
+ AtomicReference<QuorumController> value = new AtomicReference<>(null);
+ TestUtils.retryOnExceptionWithTimeout(3, 20000, () -> {
+ QuorumController activeController = null;
+ for (QuorumController controller : controllers) {
+ long curEpoch = controller.curClaimEpoch();
+ if (curEpoch != -1) {
Review comment:
To make this more intuitive, perhaps we could add a method isActive in QuorumController?
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData.Listener;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData.ListenerCollection;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metalog.LocalLogManagerTestEnv;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.controller.ConfigurationControlManagerTest.BROKER0;
+import static org.apache.kafka.controller.ConfigurationControlManagerTest.CONFIGS;
+import static org.apache.kafka.controller.ConfigurationControlManagerTest.entry;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+@Timeout(value = 40)
+public class QuorumControllerTest {
+ private static final Logger log =
+ LoggerFactory.getLogger(QuorumControllerTest.class);
+
+ /**
+ * Test creating a new QuorumController and closing it.
+ */
+ @Test
+ public void testCreateAndClose() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, __ -> { })) {
+ }
+ }
+ }
+
+ /**
+ * Test setting some configuration values and reading them back.
+ */
+ @Test
+ public void testConfigurationOperations() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, b -> b.setConfigDefs(CONFIGS))) {
+ testConfigurationOperations(controlEnv.activeController());
+ }
+ }
+ }
+
+ private void testConfigurationOperations(QuorumController controller) throws Throwable {
+ assertEquals(Collections.singletonMap(BROKER0, ApiError.NONE),
+ controller.incrementalAlterConfigs(Collections.singletonMap(
+ BROKER0, Collections.singletonMap("baz", entry(SET, "123"))), true).get());
+ assertEquals(Collections.singletonMap(BROKER0,
+ new ResultOrError<>(Collections.emptyMap())),
+ controller.describeConfigs(Collections.singletonMap(
+ BROKER0, Collections.emptyList())).get());
+ assertEquals(Collections.singletonMap(BROKER0, ApiError.NONE),
+ controller.incrementalAlterConfigs(Collections.singletonMap(
+ BROKER0, Collections.singletonMap("baz", entry(SET, "123"))), false).get());
+ assertEquals(Collections.singletonMap(BROKER0, new ResultOrError<>(Collections.
+ singletonMap("baz", "123"))),
+ controller.describeConfigs(Collections.singletonMap(
+ BROKER0, Collections.emptyList())).get());
+ }
+
+ /**
+ * Test that an incrementalAlterConfigs operation doesn't complete until the records
+ * can be written to the metadata log.
+ */
+ @Test
+ public void testDelayedConfigurationOperations() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, b -> b.setConfigDefs(CONFIGS))) {
+ testDelayedConfigurationOperations(logEnv, controlEnv.activeController());
+ }
+ }
+ }
+
+ private void testDelayedConfigurationOperations(LocalLogManagerTestEnv logEnv,
+ QuorumController controller)
+ throws Throwable {
+ logEnv.logManagers().forEach(m -> m.setMaxReadOffset(0L));
+ CompletableFuture<Map<ConfigResource, ApiError>> future1 =
+ controller.incrementalAlterConfigs(Collections.singletonMap(
+ BROKER0, Collections.singletonMap("baz", entry(SET, "123"))), false);
+ assertFalse(future1.isDone());
+ assertEquals(Collections.singletonMap(BROKER0,
+ new ResultOrError<>(Collections.emptyMap())),
+ controller.describeConfigs(Collections.singletonMap(
+ BROKER0, Collections.emptyList())).get());
+ logEnv.logManagers().forEach(m -> m.setMaxReadOffset(1L));
+ assertEquals(Collections.singletonMap(BROKER0, ApiError.NONE), future1.get());
+ }
+
+ @Test
+ public void testUnregisterBroker() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, b -> b.setConfigDefs(CONFIGS))) {
+ ListenerCollection listeners = new ListenerCollection();
+ listeners.add(new Listener().setName("PLAINTEXT").
+ setHost("localhost").setPort(9092));
+ QuorumController active = controlEnv.activeController();
+ CompletableFuture<BrokerRegistrationReply> reply = active.registerBroker(
+ new BrokerRegistrationRequestData().
+ setBrokerId(0).
+ setClusterId(Uuid.fromString("06B-K3N1TBCNYFgruEVP0Q")).
+ setIncarnationId(Uuid.fromString("kxAT73dKQsitIedpiPtwBA")).
+ setListeners(listeners));
+ assertEquals(0L, reply.get().epoch());
+ CreateTopicsRequestData createTopicsRequestData =
+ new CreateTopicsRequestData().setTopics(
+ new CreatableTopicCollection(Collections.singleton(
+ new CreatableTopic().setName("foo").setNumPartitions(1).
+ setReplicationFactor((short) 1)).iterator()));
+ assertEquals(Errors.INVALID_REPLICATION_FACTOR.code(), active.createTopics(
+ createTopicsRequestData).get().topics().find("foo").errorCode());
Review comment:
Hmm, why is a replication factor of 1 invalid?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+import java.util.function.Supplier;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships A callback which evaluates to true if the broker leads
+ * at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ Supplier<Boolean> hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantFence()) {
+ if (request.wantShutDown()) {
Review comment:
Do we allow a heartbeat request to set both the fence and wantShutDown flag?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573236986
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
Review comment:
Yes, it's a bit more efficient if the statements are nested. I will refactor this out into a separate function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579564654
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
I do think we should harmonize this, but I think it would be better to do that when we get rid of MetaLogShim. We've had a plan to get rid of the shim layer for a while but we just didn't have time to do it this week. So let's plan to do it then, if that makes sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578619727
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships True if the broker leads at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantShutDown()) {
+ if (hasLeaderships) {
+ log.info("Unfenced broker {} has requested and been granted a " +
+ "controlled shutdown.", brokerId);
+ return new BrokerControlStates(currentState, CONTROLLED_SHUTDOWN);
+ } else {
+ log.info("Unfenced broker {} has requested and been granted an " +
+ "immediate shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
Review comment:
OK, that makes sense. I will remove it from all non-singleton ISRs as well as removing it from all leaderships.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#issuecomment-781745956
> I think we need to handle preferred leader election in a special way. For example, if the assigned replicas are 1,2,3, isr is 2,3 and the current leader is 3, when doing preferred leader election, we want to keep the leader as 3 instead of changing it to 2.
Hmm, wouldn't we want to switch the leader to 2 in that case, since 2 is more preferred?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577176042
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (partition.leader == brokerId) {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ List<ApiMessageAndVersion> removeLeaderships(int brokerId) {
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = partition.chooseNewLeader(isrWithoutCurLeader, false);
Review comment:
In this function we are iterating only over the partitions that the given broker is a leader for.
( We obtained the iterator from `BrokersToIsrs#iterator(brokerId=brokerId, leadersOnly=true)` )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579458466
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
Review comment:
No special meaning. It's just a constant that can't be a valid leader. We could use -2 if that seems nicer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577143791
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
Review comment:
ok. We can make this an invalid request then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573256321
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
Review comment:
In general it doesn't make sense to wait for controlled shutdown if the broker is already fenced, because in that case its leaders have already been moved away. I will add a comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577188202
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
+ movingLeadersForShutDown = !records.isEmpty();
+ }
+ ControllerResult<BrokerHeartbeatReply> result = clusterControl.
+ processBrokerHeartbeat(request, lastCommittedOffset,
+ movingLeadersForShutDown);
+ if (movingLeadersForShutDown && !result.response().shouldShutDown()) {
+ updateShutdownOffset = true;
+ }
+ if (result.response().isFenced()) {
+ replicationControl.removeFromIsr(request.brokerId(), records);
Review comment:
It seemed safer to leave it in the ISR until it's ready to shut down for good. Also, if we take it out, it might just get re-added if it catches up... ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579497748
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
Review comment:
Ok, maybe the check can be `record.leader()< -1`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577127231
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
Review comment:
`handleEventException` handles logging exceptions thrown by events.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579463797
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
Review comment:
This should not happen during an alter isr request. We have code that checks if the alter isr request is attempting to remove the current leader from the isr, and returns an error if so. So the leader should not be changed by an alter isr request and therefore the leader epoch will not be.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578850598
##########
File path: metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.kafka.common.metadata.FeatureLevelRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.FeatureMap;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.apache.kafka.timeline.TimelineHashSet;
+
+
+public class FeatureControlManager {
+ /**
+ * The features supported by this controller's software.
+ */
+ private final Map<String, VersionRange> supportedFeatures;
+
+ /**
+ * Maps feature names to finalized version ranges.
+ */
+ private final TimelineHashMap<String, VersionRange> finalizedVersions;
+
+ /**
+ * The latest feature epoch.
+ */
+ private final TimelineHashSet<Long> epoch;
+
+ FeatureControlManager(Map<String, VersionRange> supportedFeatures,
+ SnapshotRegistry snapshotRegistry) {
+ this.supportedFeatures = supportedFeatures;
+ this.finalizedVersions = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.epoch = new TimelineHashSet<>(snapshotRegistry, 0);
+ }
+
+ ControllerResult<Map<String, ApiError>> updateFeatures(
+ Map<String, VersionRange> updates, Set<String> downgradeables,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures) {
+ TreeMap<String, ApiError> results = new TreeMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (Entry<String, VersionRange> entry : updates.entrySet()) {
+ results.put(entry.getKey(), updateFeature(entry.getKey(), entry.getValue(),
+ downgradeables.contains(entry.getKey()), brokerFeatures, records));
+ }
+ return new ControllerResult<>(records, results);
+ }
+
+ private ApiError updateFeature(String featureName,
+ VersionRange newRange,
+ boolean downgradeable,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures,
+ List<ApiMessageAndVersion> records) {
+ if (newRange.min() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The lower value for the new range cannot be less than 1.");
+ }
+ if (newRange.max() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The upper value for the new range cannot be less than 1.");
+ }
+ VersionRange localRange = supportedFeatures.get(featureName);
Review comment:
Hmm... right now, we don't have a good way of finding out what features the other controllers support. Maybe we will have to think more about this when we support rolling upgrade in kip-500.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579553251
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ /**
+ * A special value used to represent the leader for a partition with no leader.
+ */
+ public static final int NO_LEADER = -1;
+
+ /**
+ * A special value used to represent a PartitionChangeRecord that does not change the
+ * partition leader.
+ */
+ public static final int NO_LEADER_CHANGE = -2;
+
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == NO_LEADER_CHANGE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ boolean hasLeader() {
+ return leader != NO_LEADER;
+ }
+
+ int preferredReplica() {
+ return replicas.length == 0 ? NO_LEADER : replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, NO_LEADER, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ continue;
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ continue;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
Review comment:
We have NO_LEADER_CHANGE as the default for the serialized data. However, the active controller replays the PartitionChangeRecord created in memory, which defaults leader to NO_LEADER_CHANGE, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577865782
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,917 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
Review comment:
I normally do group private methods after internal classes, but in this case, it seemed better to keep them together. Otherwise you'd have to jump around a lot when reading the code. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577993008
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumControllerMetrics.java
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import com.yammer.metrics.core.Gauge;
+import com.yammer.metrics.core.Histogram;
+import com.yammer.metrics.core.MetricName;
+import com.yammer.metrics.core.MetricsRegistry;
+
+
+public final class QuorumControllerMetrics implements ControllerMetrics {
+ private final static MetricName ACTIVE_CONTROLLER_COUNT = new MetricName(
+ "kafka.controller", "KafkaController", "ActiveControllerCount", null);
+ private final static MetricName EVENT_QUEUE_TIME_MS = new MetricName(
+ "kafka.controller", "ControllerEventManager", "EventQueueTimeMs", null);
Review comment:
In the old controller, EventQueueTimeMs is used to measure the amount of time an event is sitting in the queue before being processed. There is a separate timer metric per controller state that measures the processing time per event type.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request,
+ long brokerEpoch,
+ FeatureMapAndEpoch finalizedFeatures) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ Optional<VersionRange> finalized = finalizedFeatures.map().get(feature.name());
+ if (finalized.isPresent()) {
+ if (!finalized.get().contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
Review comment:
Should this be named unregisterBroker?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships True if the broker leads at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantShutDown()) {
Review comment:
What about the case when request.wantFence() is true?
##########
File path: metadata/src/test/java/org/apache/kafka/controller/ClusterControlManagerTest.java
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Random;
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+@Timeout(value = 40)
+public class ClusterControlManagerTest {
+ @Test
+ public void testReplay() {
+ MockTime time = new MockTime(0, 0, 0);
+
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ClusterControlManager clusterControl = new ClusterControlManager(
+ new LogContext(), time, snapshotRegistry, 1000,
+ new SimpleReplicaPlacementPolicy(new Random()));
+ clusterControl.activate();
+ assertFalse(clusterControl.unfenced(0));
+
+ RegisterBrokerRecord brokerRecord = new RegisterBrokerRecord().setBrokerEpoch(100).setBrokerId(1);
+ brokerRecord.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setSecurityProtocol(SecurityProtocol.PLAINTEXT.id).
+ setPort((short) 9092).
+ setName("PLAINTEXT").
+ setHost("example.com"));
+ clusterControl.replay(brokerRecord);
+ clusterControl.checkBrokerEpoch(1, 100);
+ assertThrows(StaleBrokerEpochException.class,
+ () -> clusterControl.checkBrokerEpoch(1, 101));
+ assertThrows(StaleBrokerEpochException.class,
+ () -> clusterControl.checkBrokerEpoch(2, 100));
+ assertFalse(clusterControl.unfenced(0));
+ assertFalse(clusterControl.unfenced(1));
+
+ UnfenceBrokerRecord unfenceBrokerRecord =
+ new UnfenceBrokerRecord().setId(1).setEpoch(100);
+ clusterControl.replay(unfenceBrokerRecord);
+ assertFalse(clusterControl.unfenced(0));
+ assertTrue(clusterControl.unfenced(1));
+ }
+
+ @Test
+ public void testDecommission() throws Exception {
Review comment:
testUnregister?
##########
File path: metadata/src/test/java/org/apache/kafka/controller/ClientQuotaControlManagerTest.java
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.config.internals.QuotaConfigs;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Consumer;
+import java.util.stream.Collectors;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+@Timeout(value = 40)
+public class ClientQuotaControlManagerTest {
+
+ @Test
+ public void testInvalidEntityTypes() {
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ClientQuotaControlManager manager = new ClientQuotaControlManager(snapshotRegistry);
+
+ // Unknown type "foo"
+ assertInvalidEntity(manager, entity("foo", "bar"));
+
+ // Null type
+ assertInvalidEntity(manager, entity(null, "null"));
+
+ // Valid + unknown combo
+ assertInvalidEntity(manager, entity(ClientQuotaEntity.USER, "user-1", "foo", "bar"));
+ assertInvalidEntity(manager, entity("foo", "bar", ClientQuotaEntity.IP, "1.2.3.4"));
+
+ // Invalid combinations
+ assertInvalidEntity(manager, entity(ClientQuotaEntity.USER, "user-1", ClientQuotaEntity.IP, "1.2.3.4"));
+ assertInvalidEntity(manager, entity(ClientQuotaEntity.CLIENT_ID, "user-1", ClientQuotaEntity.IP, "1.2.3.4"));
+
+ // Empty
+ assertInvalidEntity(manager, new ClientQuotaEntity(Collections.emptyMap()));
+ }
+
+ void assertInvalidEntity(ClientQuotaControlManager manager, ClientQuotaEntity entity) {
Review comment:
Could we make it private?
##########
File path: metadata/src/main/java/org/apache/kafka/metadata/BrokerHeartbeatReply.java
##########
@@ -31,16 +31,23 @@
*/
private final boolean isFenced;
+ /**
+ * True if the heartbeat reply should tell the broker that it should shut down.
Review comment:
Hmm, the comment seems to be the same as shouldShutDown.
##########
File path: metadata/src/test/java/org/apache/kafka/controller/BrokerHeartbeatManagerTest.java
##########
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Optional;
+import java.util.Set;
+import java.util.TreeSet;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.controller.BrokerHeartbeatManager.BrokerHeartbeatState;
+import org.apache.kafka.controller.BrokerHeartbeatManager.BrokerHeartbeatStateIterator;
+import org.apache.kafka.controller.BrokerHeartbeatManager.BrokerHeartbeatStateList;
+import org.apache.kafka.controller.BrokerHeartbeatManager.UsableBrokerIterator;
+import org.apache.kafka.metadata.UsableBroker;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+@Timeout(40)
+public class BrokerHeartbeatManagerTest {
+ private static BrokerHeartbeatManager newBrokerHeartbeatManager() {
+ LogContext logContext = new LogContext();
+ MockTime time = new MockTime(0, 1_000_000, 0);
+ return new BrokerHeartbeatManager(logContext, time, 10_000_000);
+ }
+
+ @Test
+ public void testHasValidSession() {
+ BrokerHeartbeatManager manager = newBrokerHeartbeatManager();
+ MockTime time = (MockTime) manager.time();
+ assertFalse(manager.hasValidSession(0));
+ manager.touch(0, false, 0);
+ time.sleep(5);
+ manager.touch(1, false, 0);
+ manager.touch(2, false, 0);
+ assertTrue(manager.hasValidSession(0));
+ assertTrue(manager.hasValidSession(1));
+ assertTrue(manager.hasValidSession(2));
+ assertFalse(manager.hasValidSession(3));
+ time.sleep(6);
+ assertFalse(manager.hasValidSession(0));
+ assertTrue(manager.hasValidSession(1));
+ assertTrue(manager.hasValidSession(2));
+ assertFalse(manager.hasValidSession(3));
+ manager.remove(2);
+ assertFalse(manager.hasValidSession(2));
+ manager.remove(1);
+ assertFalse(manager.hasValidSession(1));
+ }
+
+ @Test
+ public void testMaybeFenceLeastRecentlyContacted() {
Review comment:
testFindStaleBrokers ?
##########
File path: metadata/src/test/java/org/apache/kafka/controller/ClusterControlManagerTest.java
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Random;
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+@Timeout(value = 40)
+public class ClusterControlManagerTest {
+ @Test
+ public void testReplay() {
+ MockTime time = new MockTime(0, 0, 0);
+
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ClusterControlManager clusterControl = new ClusterControlManager(
+ new LogContext(), time, snapshotRegistry, 1000,
+ new SimpleReplicaPlacementPolicy(new Random()));
+ clusterControl.activate();
+ assertFalse(clusterControl.unfenced(0));
+
+ RegisterBrokerRecord brokerRecord = new RegisterBrokerRecord().setBrokerEpoch(100).setBrokerId(1);
+ brokerRecord.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setSecurityProtocol(SecurityProtocol.PLAINTEXT.id).
+ setPort((short) 9092).
+ setName("PLAINTEXT").
+ setHost("example.com"));
+ clusterControl.replay(brokerRecord);
+ clusterControl.checkBrokerEpoch(1, 100);
+ assertThrows(StaleBrokerEpochException.class,
+ () -> clusterControl.checkBrokerEpoch(1, 101));
+ assertThrows(StaleBrokerEpochException.class,
+ () -> clusterControl.checkBrokerEpoch(2, 100));
+ assertFalse(clusterControl.unfenced(0));
+ assertFalse(clusterControl.unfenced(1));
+
+ UnfenceBrokerRecord unfenceBrokerRecord =
+ new UnfenceBrokerRecord().setId(1).setEpoch(100);
+ clusterControl.replay(unfenceBrokerRecord);
+ assertFalse(clusterControl.unfenced(0));
+ assertTrue(clusterControl.unfenced(1));
+ }
+
+ @Test
+ public void testDecommission() throws Exception {
+ RegisterBrokerRecord brokerRecord = new RegisterBrokerRecord().
+ setBrokerId(1).
+ setBrokerEpoch(100).
+ setIncarnationId(Uuid.fromString("fPZv1VBsRFmnlRvmGcOW9w")).
+ setRack("arack");
+ brokerRecord.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setSecurityProtocol(SecurityProtocol.PLAINTEXT.id).
+ setPort((short) 9092).
+ setName("PLAINTEXT").
+ setHost("example.com"));
+ SnapshotRegistry snapshotRegistry = new SnapshotRegistry(new LogContext());
+ ClusterControlManager clusterControl = new ClusterControlManager(
+ new LogContext(), new MockTime(0, 0, 0), snapshotRegistry, 1000,
+ new SimpleReplicaPlacementPolicy(new Random()));
+ clusterControl.activate();
+ clusterControl.replay(brokerRecord);
+ assertEquals(new BrokerRegistration(1, 100,
+ Uuid.fromString("fPZv1VBsRFmnlRvmGcOW9w"), Collections.singletonMap("PLAINTEXT",
+ new Endpoint("PLAINTEXT", SecurityProtocol.PLAINTEXT, "example.com", 9092)),
+ Collections.emptyMap(), Optional.of("arack"), true),
+ clusterControl.brokerRegistrations().get(1));
+ ControllerResult<Void> result = clusterControl.decommissionBroker(1);
+ ControllerTestUtils.replayAll(clusterControl, result.records());
+ assertFalse(clusterControl.brokerRegistrations().containsKey(1));
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {3, 10})
+ public void testChooseRandomRegistered(int numUsableBrokers) throws Exception {
Review comment:
testPlaceReplicas ?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
Review comment:
We are generating an UnregisterBrokerRecord.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships True if the broker leads at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantShutDown()) {
+ if (hasLeaderships) {
+ log.info("Unfenced broker {} has requested and been granted a " +
+ "controlled shutdown.", brokerId);
+ return new BrokerControlStates(currentState, CONTROLLED_SHUTDOWN);
+ } else {
+ log.info("Unfenced broker {} has requested and been granted an " +
+ "immediate shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ }
+ }
+ return new BrokerControlStates(currentState, UNFENCED);
+
+ case CONTROLLED_SHUTDOWN:
+ if (hasLeaderships) {
+ log.debug("Broker {} is in controlled shutdown state, but can not " +
+ "shut down because more leaders still need to be moved.", brokerId);
+ return new BrokerControlStates(currentState, CONTROLLED_SHUTDOWN);
+ }
+ long lowestActiveOffset = lowestActiveOffset();
+ if (broker.controlledShutDownOffset <= lowestActiveOffset) {
+ log.info("The request from broker {} to shut down has been granted " +
+ "since the lowest active offset {} is now greater than the " +
+ "broker's controlled shutdown offset {}.", brokerId,
+ lowestActiveOffset, broker.controlledShutDownOffset);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
Review comment:
Is there any benefit/enough to only allow the broker to shutdown when all brokers have caught up to controlledShutDownOffset? The ZK-based criteria is that a broker is allowed to shut down as long as all leaders have been moved off the shutting down broker.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ ElectLeadersResponseData response = new ElectLeadersResponseData();
+ for (TopicPartitions topic : request.topicPartitions()) {
+ ReplicaElectionResult topicResults =
+ new ReplicaElectionResult().setTopic(topic.topic());
+ response.replicaElectionResults().add(topicResults);
+ for (int partitionId : topic.partitions()) {
+ ApiError error = electLeader(topic.topic(), partitionId, unclean, records);
+ topicResults.partitionResult().add(new PartitionResult().
+ setPartitionId(partitionId).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ static boolean electionIsUnclean(byte electionType) {
+ ElectionType type;
+ try {
+ type = ElectionType.valueOf(electionType);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidRequestException("Unknown election type " + (int) electionType);
+ }
+ return type == ElectionType.UNCLEAN;
+ }
+
+ ApiError electLeader(String topic, int partitionId, boolean unclean,
+ List<ApiMessageAndVersion> records) {
+ Uuid topicId = topicsByName.get(topic);
+ if (topicId == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic as " + topic);
+ }
+ TopicControlInfo topicInfo = topics.get(topicId);
+ if (topicInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic id as " + topicId);
+ }
+ PartitionControlInfo partitionInfo = topicInfo.parts.get(partitionId);
+ if (partitionInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such partition as " + topic + "-" + partitionId);
+ }
+ int newLeader = chooseNewLeader(partitionInfo, partitionInfo.isr, unclean);
+ if (newLeader == partitionInfo.leader) {
+ if (newLeader < 0) {
+ return new ApiError(Errors.LEADER_NOT_AVAILABLE,
+ "Unable to find any leader for the partition.");
+ } else {
+ return ApiError.NONE;
+ }
+ } else {
+ int[] newIsr = partitionInfo.isr;
+ if (!Replicas.contains(partitionInfo.isr, newLeader)) {
+ newIsr = new int[] {newLeader};
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionId).
+ setTopicId(topicId).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partitionInfo.leaderEpoch + 1).
+ setPartitionEpoch(partitionInfo.partitionEpoch + 1), (short) 0));
+ return ApiError.NONE;
+ }
+ }
+
+ int chooseNewLeader(PartitionControlInfo partition, int[] newIsr, boolean unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (clusterControl.unfenced(replica)) {
+ return replica;
+ }
+ }
+ }
+ return -1;
+ }
+
+ ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request, long lastCommittedOffset) {
+ int brokerId = request.brokerId();
+ long brokerEpoch = request.brokerEpoch();
+ clusterControl.checkBrokerEpoch(brokerId, brokerEpoch);
+ BrokerHeartbeatManager heartbeatManager = clusterControl.heartbeatManager();
+ BrokerControlStates states = heartbeatManager.calculateNextBrokerState(brokerId,
+ request, lastCommittedOffset, brokersToIsrs.hasLeaderships(brokerId));
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ if (states.current() != states.next()) {
+ switch (states.next()) {
+ case FENCED:
+ handleBrokerFenced(brokerId, records);
+ break;
+ case UNFENCED:
+ handleBrokerUnfenced(brokerId, brokerEpoch, records);
+ break;
+ case CONTROLLED_SHUTDOWN:
+ removeLeaderships(brokerId, records);
Review comment:
(1) In ZK-based approach, we do leader election a bit differently for controlled shutdown. If we can't select a leader from the remaining ISR, we just leave the current leader as it is. This gives the shutting down broker a chance to retry controlled shutdown until the timeout.
(2) In ZK-based approach, we also remove the broker from isr for other partitions whose leader is not on the shutting down broker.
> It seemed safer to leave it in the ISR until it's ready to shut down for good. Also, if we take it out, it might just get re-added if it catches up... ?
That's true and is an existing problem. One way to address this is to include partitionEpoch in the follower fetch request. The leader could then reject a follower request if the partitionEpoch doesn't match. This can be done in a followup PR.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships True if the broker leads at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantShutDown()) {
+ if (hasLeaderships) {
+ log.info("Unfenced broker {} has requested and been granted a " +
+ "controlled shutdown.", brokerId);
+ return new BrokerControlStates(currentState, CONTROLLED_SHUTDOWN);
+ } else {
+ log.info("Unfenced broker {} has requested and been granted an " +
+ "immediate shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
Review comment:
If a broker wants to shut down and is only included in ISR, we still want to remove the broker from ISR before allowing it to shutdown. Otherwise, a new published record needs to wait for the session timeout before it can be committed.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
Review comment:
Could we add a TODO for handling the preferred leader election?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
Review comment:
Is this check already implied since we are iterating `brokersToIsrs`?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
It seems that using long for leaderEpoch in the log requires a bigger discussion. Could we use int for now?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
Review comment:
Are all records generated in a single ControllerWriteEvent appended to the metadata log atomically?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578851481
##########
File path: metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.kafka.common.metadata.FeatureLevelRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.FeatureMap;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.apache.kafka.timeline.TimelineHashSet;
+
+
+public class FeatureControlManager {
+ /**
+ * The features supported by this controller's software.
+ */
+ private final Map<String, VersionRange> supportedFeatures;
+
+ /**
+ * Maps feature names to finalized version ranges.
+ */
+ private final TimelineHashMap<String, VersionRange> finalizedVersions;
+
+ /**
+ * The latest feature epoch.
+ */
+ private final TimelineHashSet<Long> epoch;
+
+ FeatureControlManager(Map<String, VersionRange> supportedFeatures,
+ SnapshotRegistry snapshotRegistry) {
+ this.supportedFeatures = supportedFeatures;
+ this.finalizedVersions = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.epoch = new TimelineHashSet<>(snapshotRegistry, 0);
+ }
+
+ ControllerResult<Map<String, ApiError>> updateFeatures(
+ Map<String, VersionRange> updates, Set<String> downgradeables,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures) {
+ TreeMap<String, ApiError> results = new TreeMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (Entry<String, VersionRange> entry : updates.entrySet()) {
+ results.put(entry.getKey(), updateFeature(entry.getKey(), entry.getValue(),
+ downgradeables.contains(entry.getKey()), brokerFeatures, records));
+ }
+ return new ControllerResult<>(records, results);
+ }
+
+ private ApiError updateFeature(String featureName,
+ VersionRange newRange,
+ boolean downgradeable,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures,
+ List<ApiMessageAndVersion> records) {
+ if (newRange.min() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The lower value for the new range cannot be less than 1.");
+ }
+ if (newRange.max() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The upper value for the new range cannot be less than 1.");
+ }
+ VersionRange localRange = supportedFeatures.get(featureName);
Review comment:
Yes, that's the problem. From a consistency perspective, it seems that we should use the supported features from either all controller nodes or none.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579547404
##########
File path: metadata/src/main/resources/common/metadata/PartitionRecord.json
##########
@@ -34,6 +34,8 @@
{ "name": "Leader", "type": "int32", "versions": "0+", "default": "-1",
"about": "The lead replica, or -1 if there is no leader." },
{ "name": "LeaderEpoch", "type": "int32", "versions": "0+", "default": "-1",
- "about": "An epoch that gets incremented each time we change the ISR." }
+ "about": "An epoch that gets incremented each time we change the ISR." },
+ { "name": "PartitionEpoch", "type": "int32", "versions": "0+", "default": "-1",
Review comment:
filed KAFKA-12349
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577143791
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573238990
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
Review comment:
Good question. A fenced broker will not have leaders, so there should be no leaders to move away. More specifically, if any fenced broker tries to enter controlled shutdown, it will be shut down immediately. I'll add a comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579547731
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
Review comment:
merge only bumps the epoch if the leader was set.
```
PartitionControlInfo merge(PartitionChangeRecord record) {
int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
int newLeader;
int newLeaderEpoch;
if (record.leader() == NO_LEADER_CHANGE) {
newLeader = leader;
newLeaderEpoch = leaderEpoch;
} else {
newLeader = record.leader();
newLeaderEpoch = leaderEpoch + 1;
}
return new PartitionControlInfo(replicas,
newIsr,
removingReplicas,
addingReplicas,
newLeader,
newLeaderEpoch,
partitionEpoch + 1);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579378306
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,875 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
Review comment:
I changed this so that the leader epoch is bumped if and only if there is a leader present in the PartitionChange record. (It is possible to bump the epoch without changing the leader by including the same leader again in the record.)
We now use this during controlled shutdown to unconditionally bump the leader epochs. Otherwise, we only bump the leader epochs if the leader changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577929403
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
+ movingLeadersForShutDown = !records.isEmpty();
+ }
+ ControllerResult<BrokerHeartbeatReply> result = clusterControl.
+ processBrokerHeartbeat(request, lastCommittedOffset,
+ movingLeadersForShutDown);
+ if (movingLeadersForShutDown && !result.response().shouldShutDown()) {
+ updateShutdownOffset = true;
+ }
+ if (result.response().isFenced()) {
+ replicationControl.removeFromIsr(request.brokerId(), records);
+ }
+ rescheduleMaybeFenceReplicas();
+ records.addAll(result.records());
+ return new ControllerResult<>(records, result.response());
+ }
+
+ @Override
+ public void processBatchEndOffset(long offset) {
+ if (updateShutdownOffset) {
+ clusterControl.updateShutdownOffset(request.brokerId(), offset);
+ }
+ }
+ }
+
+ @Override
+ public CompletableFuture<BrokerHeartbeatReply>
+ processBrokerHeartbeat(BrokerHeartbeatRequestData request) {
+ return appendWriteEvent("processBrokerHeartbeat",
+ new ProcessBrokerHeartbeatOperation(request));
+ }
+
+ @Override
+ public CompletableFuture<BrokerRegistrationReply>
+ registerBroker(BrokerRegistrationRequestData request) {
+ return appendWriteEvent("registerBroker", () -> {
+ ControllerResult<BrokerRegistrationReply> result = clusterControl.
+ registerBroker(request, writeOffset + 1, featureControl.
+ finalizedFeaturesAndEpoch(Long.MAX_VALUE));
Review comment:
Thanks, @rondagostino . In the latest version, I do not remove brokers from singleton ISRs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577866850
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
Review comment:
added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578857828
##########
File path: metadata/src/test/java/org/apache/kafka/controller/QuorumControllerTest.java
##########
@@ -0,0 +1,179 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData.Listener;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData.ListenerCollection;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metalog.LocalLogManagerTestEnv;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.controller.ConfigurationControlManagerTest.BROKER0;
+import static org.apache.kafka.controller.ConfigurationControlManagerTest.CONFIGS;
+import static org.apache.kafka.controller.ConfigurationControlManagerTest.entry;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+
+@Timeout(value = 40)
+public class QuorumControllerTest {
+ private static final Logger log =
+ LoggerFactory.getLogger(QuorumControllerTest.class);
+
+ /**
+ * Test creating a new QuorumController and closing it.
+ */
+ @Test
+ public void testCreateAndClose() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, __ -> { })) {
+ }
+ }
+ }
+
+ /**
+ * Test setting some configuration values and reading them back.
+ */
+ @Test
+ public void testConfigurationOperations() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, b -> b.setConfigDefs(CONFIGS))) {
+ testConfigurationOperations(controlEnv.activeController());
+ }
+ }
+ }
+
+ private void testConfigurationOperations(QuorumController controller) throws Throwable {
+ assertEquals(Collections.singletonMap(BROKER0, ApiError.NONE),
+ controller.incrementalAlterConfigs(Collections.singletonMap(
+ BROKER0, Collections.singletonMap("baz", entry(SET, "123"))), true).get());
+ assertEquals(Collections.singletonMap(BROKER0,
+ new ResultOrError<>(Collections.emptyMap())),
+ controller.describeConfigs(Collections.singletonMap(
+ BROKER0, Collections.emptyList())).get());
+ assertEquals(Collections.singletonMap(BROKER0, ApiError.NONE),
+ controller.incrementalAlterConfigs(Collections.singletonMap(
+ BROKER0, Collections.singletonMap("baz", entry(SET, "123"))), false).get());
+ assertEquals(Collections.singletonMap(BROKER0, new ResultOrError<>(Collections.
+ singletonMap("baz", "123"))),
+ controller.describeConfigs(Collections.singletonMap(
+ BROKER0, Collections.emptyList())).get());
+ }
+
+ /**
+ * Test that an incrementalAlterConfigs operation doesn't complete until the records
+ * can be written to the metadata log.
+ */
+ @Test
+ public void testDelayedConfigurationOperations() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, b -> b.setConfigDefs(CONFIGS))) {
+ testDelayedConfigurationOperations(logEnv, controlEnv.activeController());
+ }
+ }
+ }
+
+ private void testDelayedConfigurationOperations(LocalLogManagerTestEnv logEnv,
+ QuorumController controller)
+ throws Throwable {
+ logEnv.logManagers().forEach(m -> m.setMaxReadOffset(0L));
+ CompletableFuture<Map<ConfigResource, ApiError>> future1 =
+ controller.incrementalAlterConfigs(Collections.singletonMap(
+ BROKER0, Collections.singletonMap("baz", entry(SET, "123"))), false);
+ assertFalse(future1.isDone());
+ assertEquals(Collections.singletonMap(BROKER0,
+ new ResultOrError<>(Collections.emptyMap())),
+ controller.describeConfigs(Collections.singletonMap(
+ BROKER0, Collections.emptyList())).get());
+ logEnv.logManagers().forEach(m -> m.setMaxReadOffset(1L));
+ assertEquals(Collections.singletonMap(BROKER0, ApiError.NONE), future1.get());
+ }
+
+ @Test
+ public void testUnregisterBroker() throws Throwable {
+ try (LocalLogManagerTestEnv logEnv = new LocalLogManagerTestEnv(1)) {
+ try (QuorumControllerTestEnv controlEnv =
+ new QuorumControllerTestEnv(logEnv, b -> b.setConfigDefs(CONFIGS))) {
+ ListenerCollection listeners = new ListenerCollection();
+ listeners.add(new Listener().setName("PLAINTEXT").
+ setHost("localhost").setPort(9092));
+ QuorumController active = controlEnv.activeController();
+ CompletableFuture<BrokerRegistrationReply> reply = active.registerBroker(
+ new BrokerRegistrationRequestData().
+ setBrokerId(0).
+ setClusterId(Uuid.fromString("06B-K3N1TBCNYFgruEVP0Q")).
+ setIncarnationId(Uuid.fromString("kxAT73dKQsitIedpiPtwBA")).
+ setListeners(listeners));
+ assertEquals(0L, reply.get().epoch());
+ CreateTopicsRequestData createTopicsRequestData =
+ new CreateTopicsRequestData().setTopics(
+ new CreatableTopicCollection(Collections.singleton(
+ new CreatableTopic().setName("foo").setNumPartitions(1).
+ setReplicationFactor((short) 1)).iterator()));
+ assertEquals(Errors.INVALID_REPLICATION_FACTOR.code(), active.createTopics(
+ createTopicsRequestData).get().topics().find("foo").errorCode());
Review comment:
There are no unfenced brokers (as you mentioned earlier, we should change this so that it places on the fenced broker). I will add a TODO
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577916988
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
Review comment:
that is correct. the broker must send another heartbeat before it is allowed to shut down.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579522201
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param alwaysBumpLeaderEpoch True if we should generate a record that
+ * always increments the leader epoch.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, boolean alwaysBumpLeaderEpoch,
+ List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ PartitionChangeRecord record = new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id);
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the ISR.
+ if (alwaysBumpLeaderEpoch || record.leader() != -1) {
+ // The partition is now leaderless, so set its leader to -1.
+ record.setLeader(-1);
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ }
+ } else {
+ record.setIsr(Replicas.toList(newIsr));
+ if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ int newLeader = chooseNewLeader(partition, newIsr, false);
+ record.setLeader(newLeader);
+ } else if (alwaysBumpLeaderEpoch) {
+ record.setLeader(partition.leader);
+ }
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ }
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setLeader(brokerId), (short) 0));
+ }
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ ElectLeadersResponseData response = new ElectLeadersResponseData();
+ for (TopicPartitions topic : request.topicPartitions()) {
+ ReplicaElectionResult topicResults =
+ new ReplicaElectionResult().setTopic(topic.topic());
+ response.replicaElectionResults().add(topicResults);
+ for (int partitionId : topic.partitions()) {
+ ApiError error = electLeader(topic.topic(), partitionId, unclean, records);
+ topicResults.partitionResult().add(new PartitionResult().
+ setPartitionId(partitionId).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ static boolean electionIsUnclean(byte electionType) {
+ ElectionType type;
+ try {
+ type = ElectionType.valueOf(electionType);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidRequestException("Unknown election type " + (int) electionType);
+ }
+ return type == ElectionType.UNCLEAN;
+ }
+
+ ApiError electLeader(String topic, int partitionId, boolean unclean,
+ List<ApiMessageAndVersion> records) {
+ Uuid topicId = topicsByName.get(topic);
+ if (topicId == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic as " + topic);
+ }
+ TopicControlInfo topicInfo = topics.get(topicId);
+ if (topicInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic id as " + topicId);
+ }
+ PartitionControlInfo partitionInfo = topicInfo.parts.get(partitionId);
+ if (partitionInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such partition as " + topic + "-" + partitionId);
+ }
+ int newLeader = chooseNewLeader(partitionInfo, partitionInfo.isr, unclean);
+ if (newLeader < 0) {
+ // If we can't find any leader for the partition, return an error.
+ return new ApiError(Errors.LEADER_NOT_AVAILABLE,
+ "Unable to find any leader for the partition.");
+ }
+ if (newLeader == partitionInfo.leader) {
+ // If the new leader we picked is the same as the current leader, there is
+ // nothing to do.
+ return ApiError.NONE;
+ }
+ if (partitionInfo.leader != -1 && newLeader != partitionInfo.preferredReplica()) {
Review comment:
Even in an unclean leader election, we don't want to change the leader unless we need to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578621621
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships True if the broker leads at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantShutDown()) {
Review comment:
Good question. The broker doesn't currently request fencing once it is unfenced. But for completeness, it is simple to support this and it makes the code more intuitive, so I'll add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577919102
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
Review comment:
good catch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577135430
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
Review comment:
Yes, I think that could work for partition epoch. Let's do that once we have the initial code in, though...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577217605
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
Review comment:
> This call generates a leader change record before the following fencedBroker record. Ordering wise, it seems that we should record the fencedBroker first.
Hmm... it should be OK to remove the leaderships first.
Kip-500 Controlled shutdown works this way, for example... the shutting-down broker is not actually fenced at all until all the other brokers have removed it as a leader.
Also, wouldn't it be a bit weird to be in a state where a broker is still marked as the leader for some partition, but doesn't show up in the list of brokers given in the MetadataResponse? That would happen if we put the fencing record first. I don't think clients or brokers would handle this well.
> Also, I am wondering what's the best place to trigger leader election and removing from ISR due to fenced broker. There are multiple cases when a broker can be fenced (e.g. broker controlled shutdown, broker fenced due to no heartbeat). Instead of of doing leader election and isr removal in all those cases, another option is to tigger them in a single place when a fencedBroker record is replayed.
Replaying a record cannot trigger the creation of any additional records. This would not work since the standby controllers can't create records, after all... only the active controller.
I have created a `handleBrokerFenced` function in `ReplicationControlManager` which does most of what needs to be done for fencing, though... aside from creating the actual fencing record.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577117523
##########
File path: metadata/src/main/java/org/apache/kafka/metalog/LocalLogManager.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.metalog;
+
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+
+/**
+ * The LocalLogManager is a test implementation that relies on the contents of memory.
Review comment:
Yes. I have moved it to the test directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579547825
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param alwaysBumpLeaderEpoch True if we should generate a record that
+ * always increments the leader epoch.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, boolean alwaysBumpLeaderEpoch,
Review comment:
As per our discussion outside github, let's just use the old behavior for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573239810
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
Review comment:
> only accessed from within the controller package
Ok, I was just checking. These are all fine then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573235167
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
Review comment:
Hmm, I'm not sure I understand the question....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r580411938
##########
File path: metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import org.apache.kafka.common.metadata.FeatureLevelRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.FeatureMap;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.apache.kafka.timeline.TimelineHashSet;
+
+
+public class FeatureControlManager {
+ /**
+ * The features supported by this controller's software.
+ */
+ private final Map<String, VersionRange> supportedFeatures;
+
+ /**
+ * Maps feature names to finalized version ranges.
+ */
+ private final TimelineHashMap<String, VersionRange> finalizedVersions;
+
+ /**
+ * The latest feature epoch.
+ */
+ private final TimelineHashSet<Long> epoch;
+
+ FeatureControlManager(Map<String, VersionRange> supportedFeatures,
+ SnapshotRegistry snapshotRegistry) {
+ this.supportedFeatures = supportedFeatures;
+ this.finalizedVersions = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.epoch = new TimelineHashSet<>(snapshotRegistry, 0);
+ }
+
+ ControllerResult<Map<String, ApiError>> updateFeatures(
+ Map<String, VersionRange> updates, Set<String> downgradeables,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures) {
+ TreeMap<String, ApiError> results = new TreeMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (Entry<String, VersionRange> entry : updates.entrySet()) {
+ results.put(entry.getKey(), updateFeature(entry.getKey(), entry.getValue(),
+ downgradeables.contains(entry.getKey()), brokerFeatures, records));
+ }
+ return new ControllerResult<>(records, results);
+ }
+
+ private ApiError updateFeature(String featureName,
+ VersionRange newRange,
+ boolean downgradeable,
+ Map<Integer, Map<String, VersionRange>> brokerFeatures,
+ List<ApiMessageAndVersion> records) {
+ if (newRange.min() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The lower value for the new range cannot be less than 1.");
+ }
+ if (newRange.max() <= 0) {
+ return new ApiError(Errors.INVALID_UPDATE_VERSION,
+ "The upper value for the new range cannot be less than 1.");
+ }
+ VersionRange localRange = supportedFeatures.get(featureName);
Review comment:
Do we have a jira to track this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578817611
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,597 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+import java.util.function.Supplier;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships A callback which evaluates to true if the broker leads
+ * at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ Supplier<Boolean> hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantFence()) {
+ if (request.wantShutDown()) {
Review comment:
yes, it can set both
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577187004
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
+ isFenced = true;
+ shouldShutdown = true;
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
Review comment:
I will improve the logging a bit here. I agree that we should log when a broker shuts down or is fenced, since those are major events.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573256853
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
+ if (!broker.shuttingDown()) {
+ log.trace("broker {} should not shut down because it is not in the " +
+ "shutdown state.", brokerId);
+ return false;
+ }
+ long curOffset = lowestActiveOffset();
Review comment:
I'll switch to `lowestActiveOffset`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578629900
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
Review comment:
We're iterating over the partitions with no leader, which may or may not have the newly activated broker in their ISR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r575409674
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
+ isFenced = true;
+ shouldShutdown = true;
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ } else {
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ }
+ }
+ heartbeatManager.touch(brokerId, isFenced, request.currentMetadataOffset());
+ ControllerResult<BrokerHeartbeatReply> result = new ControllerResult<>(records,
+ new BrokerHeartbeatReply(isCaughtUp, isFenced, shouldShutdown));
+ if (log.isTraceEnabled()) {
+ log.trace("processed broker heartbeat {}. lastCommittedOffset = {}, " +
+ "movingLeadersForShutDown = {}, result = {}", request,
+ lastCommittedOffset, movingLeadersForShutDown, result);
+ }
+ return result;
+ }
+
+ public void replay(RegisterBrokerRecord record) {
+ int brokerId = record.brokerId();
+ List<Endpoint> listeners = new ArrayList<>();
+ for (RegisterBrokerRecord.BrokerEndpoint endpoint : record.endPoints()) {
+ listeners.add(new Endpoint(endpoint.name(),
+ SecurityProtocol.forId(endpoint.securityProtocol()),
+ endpoint.host(), endpoint.port()));
+ }
+ Map<String, VersionRange> features = new HashMap<>();
+ for (RegisterBrokerRecord.BrokerFeature feature : record.features()) {
+ features.put(feature.name(), new VersionRange(
+ feature.minSupportedVersion(), feature.maxSupportedVersion()));
+ }
+ // Normally, all newly registered brokers start off in the fenced state.
+ // If this registration record is for a broker incarnation that was already
+ // registered, though, we preserve the existing fencing state.
+ boolean fenced = true;
+ BrokerRegistration prevRegistration = brokerRegistrations.get(brokerId);
+ if (prevRegistration != null &&
+ prevRegistration.incarnationId().equals(record.incarnationId())) {
+ fenced = prevRegistration.fenced();
+ }
+ // Update broker registrations.
+ brokerRegistrations.put(brokerId, new BrokerRegistration(brokerId,
+ record.brokerEpoch(), record.incarnationId(), listeners, features,
+ Optional.ofNullable(record.rack()), fenced));
+
+ if (prevRegistration == null) {
+ log.info("Registered new broker: {}", record);
+ } else if (prevRegistration.incarnationId().equals(record.incarnationId())) {
+ log.info("Re-registered broker incarnation: {}", record);
+ } else {
+ log.info("Re-registered broker id {}: {}", brokerId, record);
+ }
+ }
+
+ public void replay(UnregisterBrokerRecord record) {
+ int brokerId = record.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration found for that id", record.toString()));
+ } else if (registration.epoch() != record.brokerEpoch()) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration with that epoch found", record.toString()));
+ } else {
+ brokerRegistrations.remove(brokerId);
+ log.info("Unregistered broker: {}", record);
+ }
+ }
+
+ public void replay(FenceBrokerRecord record) {
+ int brokerId = record.id();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration found for that id", record.toString()));
+ } else if (registration.epoch() != record.epoch()) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration with that epoch found", record.toString()));
+ } else {
+ brokerRegistrations.put(brokerId, registration.cloneWithFencing(true));
+ log.info("Fenced broker: {}", record);
+ }
+ }
+
+ public void replay(UnfenceBrokerRecord record) {
+ int brokerId = record.id();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration found for that id", record.toString()));
+ } else if (registration.epoch() != record.epoch()) {
+ throw new RuntimeException(String.format("Unable to replay %s: no broker " +
+ "registration with that epoch found", record.toString()));
+ } else {
+ brokerRegistrations.put(brokerId, registration.cloneWithFencing(false));
Review comment:
When a broker is unfenced, some of the partitions without leader could have a new leader now. So, it seems that we need to trigger a leader election here.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokersToIsrs.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+
+
+/**
+ * Associates brokers with their in-sync partitions.
+ *
+ * This is useful when we need to remove a broker from all the ISRs, or move all leaders
+ * away from a broker.
+ *
+ * We also track all the partitions that currently have no leader.
+ *
+ * The core data structure is a map from broker IDs to topic maps. Each topic map relates
+ * topic UUIDs to arrays of partition IDs.
+ *
+ * Each entry in the array has a high bit which indicates that the broker is the leader
+ * for the given partition, as well as 31 low bits which contain the partition id. This
+ * works because partition IDs cannot be negative.
+ */
+public class BrokersToIsrs {
+ private final static int[] EMPTY = new int[0];
+
+ private final static int LEADER_FLAG = 0x8000_0000;
+
+ private final static int REPLICA_MASK = 0x7fff_ffff;
+
+ static class TopicPartition {
+ private final Uuid topicId;
+ private final int partitionId;
+
+ TopicPartition(Uuid topicId, int partitionId) {
+ this.topicId = topicId;
+ this.partitionId = partitionId;
+ }
+
+ public Uuid topicId() {
+ return topicId;
+ }
+
+ public int partitionId() {
+ return partitionId;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof TopicPartition)) return false;
+ TopicPartition other = (TopicPartition) o;
+ return other.topicId.equals(topicId) && other.partitionId == partitionId;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(topicId, partitionId);
+ }
+
+ @Override
+ public String toString() {
+ return topicId + ":" + partitionId;
+ }
+ }
+
+ static class PartitionsOnReplicaIterator implements Iterator<TopicPartition> {
+ private final Iterator<Entry<Uuid, int[]>> iterator;
+ private final boolean leaderOnly;
+ private int offset = 0;
+ Uuid uuid = Uuid.ZERO_UUID;
+ int[] replicas = EMPTY;
+ private TopicPartition next = null;
+
+ PartitionsOnReplicaIterator(Map<Uuid, int[]> topicMap, boolean leaderOnly) {
+ this.iterator = topicMap.entrySet().iterator();
+ this.leaderOnly = leaderOnly;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) return true;
+ while (true) {
+ if (offset >= replicas.length) {
+ if (!iterator.hasNext()) return false;
+ offset = 0;
+ Entry<Uuid, int[]> entry = iterator.next();
+ uuid = entry.getKey();
+ replicas = entry.getValue();
+ }
+ int replica = replicas[offset++];
+ if ((!leaderOnly) || (replica & LEADER_FLAG) != 0) {
+ next = new TopicPartition(uuid, replica & REPLICA_MASK);
+ return true;
+ }
+ }
+ }
+
+ @Override
+ public TopicPartition next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ TopicPartition result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ * Partitions with no isr members appear in this map under id -1.
+ */
+ private final TimelineHashMap<Integer, TimelineHashMap<Uuid, int[]>> isrMembers;
+
+ BrokersToIsrs(SnapshotRegistry snapshotRegistry) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.isrMembers = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Update our records of a partition's ISR.
+ *
+ * @param topicId The topic ID of the partition.
+ * @param partitionId The partition ID of the partition.
+ * @param prevIsr The previous ISR, or null if the partition is new.
+ * @param nextIsr The new ISR, or null if the partition is being removed.
+ * @param prevLeader The previous leader, or -1 if the partition had no leader.
+ * @param nextLeader The new leader, or -1 if the partition now has no leader.
+ */
+ void update(Uuid topicId, int partitionId, int[] prevIsr, int[] nextIsr,
+ int prevLeader, int nextLeader) {
+ int[] prev;
+ if (prevIsr == null) {
+ prev = EMPTY;
+ } else {
+ if (prevLeader == -1) {
+ prev = Replicas.copyWith(prevIsr, -1);
+ } else {
+ prev = Replicas.clone(prevIsr);
+ }
+ Arrays.sort(prev);
+ }
+ int[] next;
+ if (nextIsr == null) {
+ next = EMPTY;
+ } else {
+ if (nextLeader == -1) {
+ next = Replicas.copyWith(nextIsr, -1);
+ } else {
+ next = Replicas.clone(nextIsr);
+ }
+ Arrays.sort(next);
+ }
+ int i = 0, j = 0;
+ while (true) {
+ if (i == prev.length) {
+ if (j == next.length) {
+ break;
+ }
+ int newReplica = next[j];
+ add(newReplica, topicId, partitionId, newReplica == nextLeader);
+ j++;
+ } else if (j == next.length) {
+ int prevReplica = prev[i];
+ remove(prevReplica, topicId, partitionId, prevReplica == prevLeader);
+ i++;
+ } else {
+ int prevReplica = prev[i];
+ int newReplica = next[j];
+ if (prevReplica < newReplica) {
+ remove(prevReplica, topicId, partitionId, prevReplica == prevLeader);
+ i++;
+ } else if (prevReplica > newReplica) {
+ add(newReplica, topicId, partitionId, newReplica == nextLeader);
+ j++;
+ } else {
+ boolean wasLeader = prevReplica == prevLeader;
+ boolean isLeader = prevReplica == nextLeader;
+ if (wasLeader != isLeader) {
+ change(prevReplica, topicId, partitionId, wasLeader, isLeader);
+ }
+ i++;
+ j++;
+ }
+ }
+ }
+ }
+
+ private void add(int brokerId, Uuid topicId, int newPartition, boolean leader) {
+ if (leader) {
+ newPartition = newPartition | LEADER_FLAG;
+ }
+ TimelineHashMap<Uuid, int[]> topicMap = isrMembers.get(brokerId);
+ if (topicMap == null) {
+ topicMap = new TimelineHashMap<>(snapshotRegistry, 0);
+ isrMembers.put(brokerId, topicMap);
+ }
+ int[] partitions = topicMap.get(topicId);
+ int[] newPartitions;
+ if (partitions == null) {
+ newPartitions = new int[1];
+ } else {
+ newPartitions = new int[partitions.length + 1];
+ System.arraycopy(partitions, 0, newPartitions, 0, partitions.length);
+ }
+ newPartitions[newPartitions.length - 1] = newPartition;
+ topicMap.put(topicId, newPartitions);
+ }
+
+ private void change(int brokerId, Uuid topicId, int partition,
+ boolean wasLeader, boolean isLeader) {
+ TimelineHashMap<Uuid, int[]> topicMap = isrMembers.get(brokerId);
+ if (topicMap == null) {
+ throw new RuntimeException("Broker " + brokerId + " has no isrMembers " +
+ "entry, so we can't change " + topicId + ":" + partition);
+ }
+ int[] partitions = topicMap.get(topicId);
+ if (partitions == null) {
+ throw new RuntimeException("Broker " + brokerId + " has no " +
+ "entry in isrMembers for topic " + topicId);
+ }
+ int[] newPartitions = new int[partitions.length];
+ int target = wasLeader ? partition | LEADER_FLAG : partition;
+ for (int i = 0; i < partitions.length; i++) {
+ int cur = partitions[i];
+ if (cur == target) {
+ newPartitions[i] = isLeader ? partition | LEADER_FLAG : partition;
Review comment:
Could we just update `partitions` in place?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
Review comment:
In the ZK based logic, on receiving a controlled shutdown request, the controller tries to move the leaders off the broker. If this is successful, the controller sends a successful return for the broker to proceed with shutdown. Here, it seems that the controller will initially return shouldShutDown as false to the broker if there are leaders moved off the broker and require the broker to heartbeat again to be able to shut down.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
Review comment:
This should tigger leader election too, right?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
Review comment:
Sometimes, we update the in-memory state after the record is appended to the log. Here, it seems that we do the reverse. Should we make that consistent?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/Controller.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+
+public interface Controller extends AutoCloseable {
+ /**
+ * Change partition ISRs.
+ *
+ * @param request The AlterIsrRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request);
+
+ /**
+ * Create a batch of topics.
+ *
+ * @param request The CreateTopicsRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request);
+
+ /**
+ * Decommission a broker.
+ *
+ * @param brokerId The broker id to decommission.
+ *
+ * @return A future that is completed successfully hwne the broker is
+ * decommissioned, or if it is not registered in the first place.
+ */
+ CompletableFuture<Void> decommissionBroker(int brokerId);
Review comment:
Should this be changed to unregisterBroker to match the KIP?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
Review comment:
This call generates a leader change record before the following fencedBroker record. Ordering wise, it seems that we should record the fencedBroker first.
Also, I am wondering what's the best place to trigger leader election and removing from ISR due to fenced broker. There are multiple cases when a broker can be fenced (e.g. broker controlled shutdown, broker fenced due to no heartbeat). Instead of of doing leader election and isr removal in all those cases, another option is to tigger them in a single place when a fencedBroker record is replayed.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
Review comment:
Do we need to use `this`? Ditto below.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
Review comment:
Is the intention of `shouldShutDown` to wait until the metadata of the shutting down broker is received by other brokers? If so, not sure why `beginBrokerShutDown` sets `broker.shutDownOffset` to either MAX_LONG or 0.
Also, if `shouldShutDown` returns false, when do we get another opportunity to mark the shutting down broker as fenced?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/Controller.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+
+public interface Controller extends AutoCloseable {
+ /**
+ * Change partition ISRs.
+ *
+ * @param request The AlterIsrRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request);
+
+ /**
+ * Create a batch of topics.
+ *
+ * @param request The CreateTopicsRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request);
+
+ /**
+ * Decommission a broker.
+ *
+ * @param brokerId The broker id to decommission.
+ *
+ * @return A future that is completed successfully hwne the broker is
+ * decommissioned, or if it is not registered in the first place.
+ */
+ CompletableFuture<Void> decommissionBroker(int brokerId);
+
+ /**
+ * Describe the current configuration of various resources.
+ *
+ * @param resources A map from resources to the collection of config keys that we
+ * want to describe for each. If the collection is empty, then
+ * all configuration keys will be described.
+ *
+ * @return
+ */
+ CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources);
+
+ /**
+ * Elect new partition leaders.
+ *
+ * @param request The request.
+ *
+ * @return A future yielding the elect leaders response.
+ */
+ CompletableFuture<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request);
+
+ /**
+ * Get the current finalized feature ranges for each feature.
+ *
+ * @return A future yielding the feature ranges.
+ */
+ CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures();
+
+ /**
+ * Perform some incremental configuration changes.
+ *
+ * @param configChanges The changes.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
+ */
+ CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Map.Entry<AlterConfigOp.OpType, String>>> configChanges,
+ boolean validateOnly);
+
+ /**
+ * Perform some configuration changes using the legacy API.
+ *
+ * @param newConfigs The new configuration maps to apply.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
+ */
+ CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly);
+
+ /**
+ * Process a heartbeat from a broker.
+ *
+ * @param request The broker heartbeat request.
+ *
+ * @return A future yielding a heartbeat reply.
+ */
+ CompletableFuture<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request);
+
+ /**
+ * Attempt to register the given broker.
+ *
+ * @param request The registration request.
+ *
+ * @return A future yielding a registration reply.
+ */
+ CompletableFuture<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request);
+
+ /**
+ * Wait for the given number of brokers to be registered and unfenced.
+ * This is for testing.
+ *
+ * @param minBrokers The minimum number of brokers to wait for.
+ * @return A future which is completed when the given number of brokers
+ * is reached.
+ */
+ CompletableFuture<Void> waitForReadyBrokers(int minBrokers);
+
+ /**
+ * Perform some client quota changes
+ *
+ * @param quotaAlterations The list of quotas to alter
+ * @param validateOnly True if we should validate the changes but not apply them.
+ * @return A future yielding a map of quota entities to error results.
+ */
+ CompletableFuture<Map<ClientQuotaEntity, ApiError>> alterClientQuotas(
+ Collection<ClientQuotaAlteration> quotaAlterations, boolean validateOnly
+ );
+
+ /**
+ * Begin shutting down, but don't block. You must still call close to clean up all
+ * resources.
+ */
+ void beginShutdown();
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1.
+ */
+ long curClaimEpoch();
+
+ /**
+ * Blocks until we have shut down and freed all resources.
+ */
+ void close() throws InterruptedException;
Review comment:
Should we call this sth like waitForShutdownComplete to match `beginShutdown`?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,519 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.shutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ /**
+ * Move a broker into shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param deferred If this is true, the broker can shut down immediately.
Review comment:
It seems the broker can shut down immediately if this is false?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/LeaderAndIsr.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.List;
+
+public class LeaderAndIsr {
+ private final int leaderId;
+ private final int leaderEpoch;
+ private final List<Integer> isr;
+ private final int currentZkVersion;
Review comment:
Is this still needed with Raft metadata?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
Review comment:
Should we also trigger leader elections here? Also, should we allow broker decommission when it still has replicas?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,519 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.shutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ /**
+ * Move a broker into shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param deferred If this is true, the broker can shut down immediately.
+ * Otherwise, the broker must wait for some replicas to be moved.
+ */
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutDownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ // If the broker is fenced, we don't wait for controlled shutdown, since
+ // its leaders have already been moved away.
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
+ if (!broker.shuttingDown()) {
+ log.trace("broker {} should not shut down because it is not in the " +
+ "shutdown state.", brokerId);
+ return false;
+ }
+ long lowestActiveOffset = lowestActiveOffset();
+ if (lowestActiveOffset >= broker.shutDownOffset) {
+ log.debug("broker {} should shut down. The current lowest active offset, {} " +
+ "is greater or equal to the broker shutdown offset of {}", brokerId,
+ lowestActiveOffset, broker.shutDownOffset);
+ return true;
+ } else {
+ log.debug("broker {} must wait to shut down. The current lowest active offset, {} " +
+ "is lower than the broker shutdown offset of {}", brokerId,
+ lowestActiveOffset, broker.shutDownOffset);
+ return false;
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Update the offset at which a broker should shut down.
+ *
+ * The controller invokes this function after it has scheduled a batch of records to
+ * be written to move leaders off of the broker.
+ *
+ * @param brokerId The broker id.
+ * @param offset The new offset.
+ */
+ void updateShutdownOffset(int brokerId, long offset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ active.remove(broker);
+ broker.shutDownOffset = offset;
+ log.debug("Updated the shutdown offset for broker {} to {}.", brokerId, offset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Fenced the broker which was least recently contacted, if needed.
+ *
+ * @return -1 if no broker was fenced; the broker ID otherwise. If the return
+ * value is non-negative, this function should be called again.
+ */
+ int maybeFenceLeastRecentlyContacted() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null || hasValidSession(broker)) {
+ return -1;
+ } else {
+ unfenced.remove(broker);
+ return broker.id();
+ }
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
Review comment:
I thought we want to allow topics to be created even when there is not enough live brokers now?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
Review comment:
Should we trigger the logic for leader election/isr removal since there could be unhandled fencedBroker records when the new controller takes over?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/LeaderAndIsr.java
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.List;
+
+public class LeaderAndIsr {
Review comment:
This class seems unused.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (partition.leader == brokerId) {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ List<ApiMessageAndVersion> removeLeaderships(int brokerId) {
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = partition.chooseNewLeader(isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
Review comment:
Since we use IsrChangeRecord for changing the leader too, could we name it to sth more general?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
Review comment:
What triggers this on a hard controller failure?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
Review comment:
Could we add some comment for this class?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/PartitionLeaderElectionResult.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.requests.ApiError;
+
+public class PartitionLeaderElectionResult {
Review comment:
This class seems unused.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicaPlacementPolicy.java
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.metadata.UsableBroker;
+
+import java.util.Iterator;
+import java.util.List;
+
+
+/**
+ * Represents a deferred event in the controller purgatory.
Review comment:
Inaccurate comment.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
Review comment:
It might be useful to log both the old and the new value.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
Review comment:
Hmm, should we return the future from `appendWriteEvent` ?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
Review comment:
Are we deprecating the state-change log and the controller log that we had before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578625531
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ ElectLeadersResponseData response = new ElectLeadersResponseData();
+ for (TopicPartitions topic : request.topicPartitions()) {
+ ReplicaElectionResult topicResults =
+ new ReplicaElectionResult().setTopic(topic.topic());
+ response.replicaElectionResults().add(topicResults);
+ for (int partitionId : topic.partitions()) {
+ ApiError error = electLeader(topic.topic(), partitionId, unclean, records);
+ topicResults.partitionResult().add(new PartitionResult().
+ setPartitionId(partitionId).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ static boolean electionIsUnclean(byte electionType) {
+ ElectionType type;
+ try {
+ type = ElectionType.valueOf(electionType);
+ } catch (IllegalArgumentException e) {
+ throw new InvalidRequestException("Unknown election type " + (int) electionType);
+ }
+ return type == ElectionType.UNCLEAN;
+ }
+
+ ApiError electLeader(String topic, int partitionId, boolean unclean,
+ List<ApiMessageAndVersion> records) {
+ Uuid topicId = topicsByName.get(topic);
+ if (topicId == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic as " + topic);
+ }
+ TopicControlInfo topicInfo = topics.get(topicId);
+ if (topicInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such topic id as " + topicId);
+ }
+ PartitionControlInfo partitionInfo = topicInfo.parts.get(partitionId);
+ if (partitionInfo == null) {
+ return new ApiError(Errors.UNKNOWN_TOPIC_OR_PARTITION,
+ "No such partition as " + topic + "-" + partitionId);
+ }
+ int newLeader = chooseNewLeader(partitionInfo, partitionInfo.isr, unclean);
+ if (newLeader == partitionInfo.leader) {
+ if (newLeader < 0) {
+ return new ApiError(Errors.LEADER_NOT_AVAILABLE,
+ "Unable to find any leader for the partition.");
+ } else {
+ return ApiError.NONE;
+ }
+ } else {
+ int[] newIsr = partitionInfo.isr;
+ if (!Replicas.contains(partitionInfo.isr, newLeader)) {
+ newIsr = new int[] {newLeader};
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionId).
+ setTopicId(topicId).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partitionInfo.leaderEpoch + 1).
+ setPartitionEpoch(partitionInfo.partitionEpoch + 1), (short) 0));
+ return ApiError.NONE;
+ }
+ }
+
+ int chooseNewLeader(PartitionControlInfo partition, int[] newIsr, boolean unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean) {
+ for (int i = 0; i < partition.replicas.length; i++) {
+ int replica = partition.replicas[i];
+ if (clusterControl.unfenced(replica)) {
+ return replica;
+ }
+ }
+ }
+ return -1;
+ }
+
+ ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request, long lastCommittedOffset) {
+ int brokerId = request.brokerId();
+ long brokerEpoch = request.brokerEpoch();
+ clusterControl.checkBrokerEpoch(brokerId, brokerEpoch);
+ BrokerHeartbeatManager heartbeatManager = clusterControl.heartbeatManager();
+ BrokerControlStates states = heartbeatManager.calculateNextBrokerState(brokerId,
+ request, lastCommittedOffset, brokersToIsrs.hasLeaderships(brokerId));
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ if (states.current() != states.next()) {
+ switch (states.next()) {
+ case FENCED:
+ handleBrokerFenced(brokerId, records);
+ break;
+ case UNFENCED:
+ handleBrokerUnfenced(brokerId, brokerEpoch, records);
+ break;
+ case CONTROLLED_SHUTDOWN:
+ removeLeaderships(brokerId, records);
Review comment:
As I mentioned above, I changed it so that it now removes the broker from all non-singleton ISRs, as well as removing it from leaderships.
It seems like the remaining behavioral difference is that the new code will, if no other leader can be chosen, set the leader to -1 (offline). If we don't do this, controlled shutdown easily gets stuck if there are any partitions with replication factor = 1. Maybe we can tune this a bit later?
I like the idea of including the partition epoch in the follower fetch request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577163457
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
Review comment:
I changed it to `removeFromIsrAndLeaderships`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578876029
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureMapAndEpoch;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
Review comment:
Could we add some comments to this class?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
Review comment:
> > I think we need to handle preferred leader election in a special way. For example, if the assigned replicas are 1,2,3, isr is 2,3 and the current leader is 3, when doing preferred leader election, we want to keep the leader as 3 instead of changing it to 2.
>
> Hmm, wouldn't we want to switch the leader to 2 in that case, since 2 is more preferred?
Well, currently the contract is just that if every broker picks the preferred replica (i.e. 1st replica), the leaders will be balanced among brokers. If not, all other replicas are equivalent. Moving leaders among non-preferred replicas just creates churns without benefiting the balance.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,875 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
Review comment:
As Jason pointed out, in ZK based approach, the controller bumps up the leader epoch for removing replica from ISR too.
Also, since the broker is no longer receiving the leaderAndIsr requests, we need some logic for the broker to ignore the new partition record (for follower fetching) once it starts the controlled shutdown process.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573234880
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
Review comment:
It doesn't need to be public because it is only invoked from within the controller package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578630655
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,585 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+import static org.apache.kafka.controller.BrokerControlState.FENCED;
+import static org.apache.kafka.controller.BrokerControlState.CONTROLLED_SHUTDOWN;
+import static org.apache.kafka.controller.BrokerControlState.SHUTDOWN_NOW;
+import static org.apache.kafka.controller.BrokerControlState.UNFENCED;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long controlledShutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.controlledShutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return controlledShutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(BrokerHeartbeatManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Mark a broker as fenced.
+ *
+ * @param brokerId The ID of the broker to mark as fenced.
+ */
+ void fence(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Remove a broker.
+ *
+ * @param brokerId The ID of the broker to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.controlledShutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ /**
+ * Mark a broker as being in the controlled shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param controlledShutDownOffset The offset at which controlled shutdown will be complete.
+ */
+ void updateControlledShutdownOffset(int brokerId, long controlledShutDownOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ throw new RuntimeException("Fenced brokers cannot enter controlled shutdown.");
+ }
+ active.remove(broker);
+ broker.controlledShutDownOffset = controlledShutDownOffset;
+ log.debug("Updated the controlled shutdown offset for broker {} to {}.",
+ brokerId, controlledShutDownOffset);
+ }
+
+ /**
+ * Return the time in monotonic nanoseconds at which we should check if a broker
+ * session needs to be expired.
+ */
+ long nextCheckTimeNs() {
+ BrokerHeartbeatState broker = unfenced.first();
+ if (broker == null) {
+ return Long.MAX_VALUE;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs;
+ }
+ }
+
+ /**
+ * Find the stale brokers which haven't heartbeated in a long time, and which need to
+ * be fenced.
+ *
+ * @return A list of node IDs.
+ */
+ List<Integer> findStaleBrokers() {
+ List<Integer> nodes = new ArrayList<>();
+ BrokerHeartbeatStateIterator iterator = unfenced.iterator();
+ while (iterator.hasNext()) {
+ BrokerHeartbeatState broker = iterator.next();
+ if (hasValidSession(broker)) {
+ break;
+ }
+ nodes.add(broker.id);
+ }
+ return nodes;
+ }
+
+ /**
+ * Place replicas on unfenced brokers.
+ *
+ * @param numPartitions The number of partitions to place.
+ * @param numReplicas The number of replicas for each partition.
+ * @param idToRack A function mapping broker id to broker rack.
+ * @param policy The replica placement policy to use.
+ *
+ * @return A list of replica lists.
+ *
+ * @throws InvalidReplicationFactorException If too many replicas were requested.
+ */
+ List<List<Integer>> placeReplicas(int numPartitions, short numReplicas,
+ Function<Integer, Optional<String>> idToRack,
+ ReplicaPlacementPolicy policy) {
+ // TODO: support using fenced brokers here if necessary to get to the desired
+ // number of replicas. We probably need to add a fenced boolean in UsableBroker.
+ Iterator<UsableBroker> iterator = new UsableBrokerIterator(
+ unfenced.iterator(), idToRack);
+ return policy.createPlacement(numPartitions, numReplicas, iterator);
+ }
+
+ static class UsableBrokerIterator implements Iterator<UsableBroker> {
+ private final Iterator<BrokerHeartbeatState> iterator;
+ private final Function<Integer, Optional<String>> idToRack;
+ private UsableBroker next;
+
+ UsableBrokerIterator(Iterator<BrokerHeartbeatState> iterator,
+ Function<Integer, Optional<String>> idToRack) {
+ this.iterator = iterator;
+ this.idToRack = idToRack;
+ this.next = null;
+ }
+
+ @Override
+ public boolean hasNext() {
+ if (next != null) {
+ return true;
+ }
+ BrokerHeartbeatState result;
+ do {
+ if (!iterator.hasNext()) {
+ return false;
+ }
+ result = iterator.next();
+ } while (result.shuttingDown());
+ Optional<String> rack = idToRack.apply(result.id());
+ next = new UsableBroker(result.id(), rack);
+ return true;
+ }
+
+ @Override
+ public UsableBroker next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ UsableBroker result = next;
+ next = null;
+ return result;
+ }
+ }
+
+ BrokerControlState currentBrokerState(BrokerHeartbeatState broker) {
+ if (broker.shuttingDown()) {
+ return CONTROLLED_SHUTDOWN;
+ } else if (broker.fenced()) {
+ return FENCED;
+ } else {
+ return UNFENCED;
+ }
+ }
+
+ /**
+ * Calculate the next broker state for a broker that just sent a heartbeat request.
+ *
+ * @param brokerId The broker id.
+ * @param request The incoming heartbeat request.
+ * @param lastCommittedOffset The last committed offset of the quorum controller.
+ * @param hasLeaderships True if the broker leads at least one partition.
+ *
+ * @return The current and next broker states.
+ */
+ BrokerControlStates calculateNextBrokerState(int brokerId,
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean hasLeaderships) {
+ BrokerHeartbeatState broker = brokers.getOrDefault(brokerId,
+ new BrokerHeartbeatState(brokerId));
+ BrokerControlState currentState = currentBrokerState(broker);
+ switch (currentState) {
+ case FENCED:
+ if (request.wantShutDown()) {
+ log.info("Fenced broker {} has requested and been granted an immediate " +
+ "shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ } else if (!request.wantFence()) {
+ if (request.currentMetadataOffset() >= lastCommittedOffset) {
+ log.info("The request from broker {} to unfence has been granted " +
+ "because it has caught up with the last committed metadata " +
+ "offset {}.", brokerId, lastCommittedOffset);
+ return new BrokerControlStates(currentState, UNFENCED);
+ } else {
+ if (log.isDebugEnabled()) {
+ log.debug("The request from broker {} to unfence cannot yet " +
+ "be granted because it has not caught up with the last " +
+ "committed metadata offset {}. It is still at offset {}.",
+ brokerId, lastCommittedOffset, request.currentMetadataOffset());
+ }
+ return new BrokerControlStates(currentState, FENCED);
+ }
+ }
+ return new BrokerControlStates(currentState, FENCED);
+
+ case UNFENCED:
+ if (request.wantShutDown()) {
+ if (hasLeaderships) {
+ log.info("Unfenced broker {} has requested and been granted a " +
+ "controlled shutdown.", brokerId);
+ return new BrokerControlStates(currentState, CONTROLLED_SHUTDOWN);
+ } else {
+ log.info("Unfenced broker {} has requested and been granted an " +
+ "immediate shutdown.", brokerId);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
+ }
+ }
+ return new BrokerControlStates(currentState, UNFENCED);
+
+ case CONTROLLED_SHUTDOWN:
+ if (hasLeaderships) {
+ log.debug("Broker {} is in controlled shutdown state, but can not " +
+ "shut down because more leaders still need to be moved.", brokerId);
+ return new BrokerControlStates(currentState, CONTROLLED_SHUTDOWN);
+ }
+ long lowestActiveOffset = lowestActiveOffset();
+ if (broker.controlledShutDownOffset <= lowestActiveOffset) {
+ log.info("The request from broker {} to shut down has been granted " +
+ "since the lowest active offset {} is now greater than the " +
+ "broker's controlled shutdown offset {}.", brokerId,
+ lowestActiveOffset, broker.controlledShutDownOffset);
+ return new BrokerControlStates(currentState, SHUTDOWN_NOW);
Review comment:
Basically it lets us know that the other brokers have successfully taken over as leader (where needed) which avoids having a period of unavailability, ideally
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577929118
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
Review comment:
I don't think the state change log can scale to the number of partitions we need. It gets too verbose. Also, this information is available in the metadata log itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577178197
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577914392
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (partition.leader == brokerId) {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ List<ApiMessageAndVersion> removeLeaderships(int brokerId) {
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = partition.chooseNewLeader(isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
Review comment:
I have renamed it to `PartitionChangeRecord`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe merged pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe merged pull request #10070:
URL: https://github.com/apache/kafka/pull/10070
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577146421
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
Review comment:
Hmm, this comment shows up for me as being left in the `decommissionBroker` function, which is called in response to the decomission broker RPC, not the heartbeat RPC. Maybe this is a Github UI issue? Did you mean to leave this comment for a different function?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577176042
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (partition.leader == brokerId) {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ List<ApiMessageAndVersion> removeLeaderships(int brokerId) {
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = partition.chooseNewLeader(isrWithoutCurLeader, false);
Review comment:
In this function we are iterating only over the partitions that the given broker is a leader for.
( We called `BrokersToIsrs(brokerId=brokerId, leadersOnly=true)` )
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579547910
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
Review comment:
I added a constant. I think it looks a little nicer...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577117523
##########
File path: metadata/src/main/java/org/apache/kafka/metalog/LocalLogManager.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.metalog;
+
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+
+/**
+ * The LocalLogManager is a test implementation that relies on the contents of memory.
Review comment:
Yes, it is. I will move it to the test directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579463797
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
Review comment:
Hmm... ReplicationControlManager should not allow this to happen during an alter isr request. There is some code that checks if the alter isr request is attempting to remove the current leader from the isr, and returns an error if so. So the leader should not be changed by an alter isr request and therefore the leader epoch will not be.
```
if (!Replicas.contains(newIsr, partition.leader)) {
// An alterIsr request can't remove the current leader.
responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
setPartitionIndex(partitionData.partitionIndex()).
setErrorCode(Errors.INVALID_REQUEST.code()));
continue;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579462462
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,894 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == Integer.MIN_VALUE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ int preferredReplica() {
+ if (replicas.length == 0) return -1;
+ return replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate an
+ * UnregisterBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, false, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ * It is removed as the leader for all partitions it leads.
+ *
+ * @param brokerId The broker id.
+ * @param alwaysBumpLeaderEpoch True if we should generate a record that
+ * always increments the leader epoch.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, boolean alwaysBumpLeaderEpoch,
Review comment:
Hmm... I don't completely understand why we would want to bump the leader epoch when the controller removes a non-leader broker B but not when an AlterIsrRequest removes a non-leader broker B from the ISR. It seems like we should either bump in both scenarios or neither.
Is the fact that we bump in the first scenario just an artifact of the fact that otherwise we could not send out a leader and isr request that had a new epoch and thereby caused a change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577148212
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (partition.leader == brokerId) {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
Review comment:
I believe we should surround this section of code with the following to be sure we never drop the last ISR member:
```
// never remove the last ISR member
if (partition.isr.length > 1) {
int[] newIsr = ...
etc...
}
```
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
+ movingLeadersForShutDown = !records.isEmpty();
+ }
+ ControllerResult<BrokerHeartbeatReply> result = clusterControl.
+ processBrokerHeartbeat(request, lastCommittedOffset,
+ movingLeadersForShutDown);
+ if (movingLeadersForShutDown && !result.response().shouldShutDown()) {
+ updateShutdownOffset = true;
+ }
+ if (result.response().isFenced()) {
+ replicationControl.removeFromIsr(request.brokerId(), records);
+ }
+ rescheduleMaybeFenceReplicas();
+ records.addAll(result.records());
+ return new ControllerResult<>(records, result.response());
+ }
+
+ @Override
+ public void processBatchEndOffset(long offset) {
+ if (updateShutdownOffset) {
+ clusterControl.updateShutdownOffset(request.brokerId(), offset);
+ }
+ }
+ }
+
+ @Override
+ public CompletableFuture<BrokerHeartbeatReply>
+ processBrokerHeartbeat(BrokerHeartbeatRequestData request) {
+ return appendWriteEvent("processBrokerHeartbeat",
+ new ProcessBrokerHeartbeatOperation(request));
+ }
+
+ @Override
+ public CompletableFuture<BrokerRegistrationReply>
+ registerBroker(BrokerRegistrationRequestData request) {
+ return appendWriteEvent("registerBroker", () -> {
+ ControllerResult<BrokerRegistrationReply> result = clusterControl.
+ registerBroker(request, writeOffset + 1, featureControl.
+ finalizedFeaturesAndEpoch(Long.MAX_VALUE));
Review comment:
I believe after this we need to invoke something to cover the case where a topic has this broker as its only ISR member:
```
replicationControl.maybeMakeLeader(request.brokerId(), result.records());
```
The implementation might look like this:
```
void maybeMakeLeader(int brokerId, List<ApiMessageAndVersion> records) {
Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
while (iterator.hasNext()) {
TopicPartition topicPartition = iterator.next();
TopicControlInfo topic = topics.get(topicPartition.topicId());
if (topic == null) {
throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
"isrMembers, but not in the topics map.");
}
PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
if (partition == null) {
throw new RuntimeException("Partition " + topicPartition +
" existed in isrMembers, but not in the partitions map.");
}
// make it the leader if it is the only ISR member
if (partition.isr.length == 1 && partition.isr[0] == brokerId && partition.leader != brokerId) {
int newLeader = brokerId;
int newLeaderEpoch = partition.leaderEpoch + 1;
IsrChangeRecord message = new IsrChangeRecord().
setPartitionId(topicPartition.partitionId()).
setTopicId(topic.id).
setIsr(Replicas.toList(partition.isr)).
setLeader(newLeader).
setLeaderEpoch(newLeaderEpoch).
setPartitionEpoch(partition.partitionEpoch + 1);
records.add(new ApiMessageAndVersion(message, (short) 0));
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577920156
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,519 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutDownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutDownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutDownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ untrack(broker);
+ }
+ }
+
+ /**
+ * Stop tracking the broker in the unfenced list and active set, if it was tracked
+ * in either of these.
+ *
+ * @param broker The broker state to stop tracking.
+ */
+ private void untrack(BrokerHeartbeatState broker) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ // Remove the broker from the unfenced list and/or the active set. Its
+ // position in either of those data structures depends on values we are
+ // changing here. We will re-add it if necessary at the end of this function.
+ untrack(broker);
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ // If a broker is fenced, it leaves controlled shutdown. On its next heartbeat,
+ // it will shut down immediately.
+ broker.shutDownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ /**
+ * Move a broker into shutdown state.
+ *
+ * @param brokerId The broker id.
+ * @param deferred If this is true, the broker can shut down immediately.
Review comment:
this got refactored. hopefully it is clearer now. yes, a broker can shut down immediately in some cases
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573254808
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
Review comment:
If it's already shutting down nothing happens
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
Review comment:
package-private is OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577879960
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,917 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
Review comment:
Sounds good too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577915565
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
Review comment:
> Should we also trigger leader elections here?
Good catch. Yes, this should be moving leaders. I fixed this in the latest version of the PR.
> Also, should we allow broker decommission when it still has replicas?
Yes, this is allowed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578065045
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
Review comment:
Are all records generated in a single ControllerWriteEvent appended to the metadata log atomically?
colin:
Another comment without a reply link. I will reply inline.
RIght now the answer is yes. Eventually we plan on supporting multiple batches.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577259797
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
Review comment:
Sounds good. Could we add a comment to make that clear?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577916179
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
Review comment:
I restructured this code a bit. But yes, it does move leaders if needed (it's clearer in the new version I think)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r577868843
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
Review comment:
We can get here just because the user made an invalid RPC, so I don't know if WARN is appropriate. I'll change it to INFO for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578003900
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
It seems that using long for leaderEpoch in the log requires a bigger discussion. Could we use int for now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r578627766
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,900 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(partition.leader).
+ setLeaderEpoch(partition.leaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being fenced.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+
+ void handleBrokerFenced(int brokerId, List<ApiMessageAndVersion> records) {
+ BrokerRegistration brokerRegistration = clusterControl.brokerRegistrations().get(brokerId);
+ if (brokerRegistration == null) {
+ throw new RuntimeException("Can't find broker registration for broker " + brokerId);
+ }
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerRegistration.epoch()), (short) 0));
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker being unregistered.
+ *
+ * First, we remove this broker from any non-singleton ISR. Then we generate a
+ * FenceBrokerRecord.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnregistered(int brokerId, long brokerEpoch,
+ List<ApiMessageAndVersion> records) {
+ handleNodeDeactivated(brokerId, records);
+ records.add(new ApiMessageAndVersion(new UnregisterBrokerRecord().
+ setBrokerId(brokerId).setBrokerEpoch(brokerEpoch), (short) 0));
+ }
+
+ /**
+ * Handle a broker being deactivated. This means we remove it from any ISR that has
+ * more than one element. We do not remove the broker from ISRs where it is the only
+ * member since this would preclude clean leader election in the future.
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeDeactivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (newIsr.length == 0) {
+ // We don't want to shrink the ISR to size 0. So, leave the node in the
+ // ISR, but set the leader to -1 (no leader).
+ newIsr = partition.isr;
+ newLeader = -1;
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else if (partition.leader == brokerId) {
+ // The fenced node will no longer be the leader.
+ newLeader = chooseNewLeader(partition, newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ // The fenced node wasn't the leader, so no leader change is needed.
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ /**
+ * Generate the appropriate records to handle a broker becoming unfenced.
+ *
+ * First, we create an UnfenceBrokerRecord. Then, we check if if there are any
+ * partitions that don't currently have a leader that should be led by the newly
+ * unfenced broker.
+ *
+ * @param brokerId The broker id.
+ * @param brokerEpoch The broker epoch.
+ * @param records The record list to append to.
+ */
+ void handleBrokerUnfenced(int brokerId, long brokerEpoch, List<ApiMessageAndVersion> records) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(brokerEpoch), (short) 0));
+ handleNodeActivated(brokerId, records);
+ }
+
+ /**
+ * Handle a broker being activated. This means we check if it can become the leader
+ * for any partition that currently has no leader (aka offline partition).
+ *
+ * @param brokerId The broker id.
+ * @param records The record list to append to.
+ */
+ void handleNodeActivated(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.noLeaderIterator();
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ // TODO: if this partition is configured for unclean leader election,
+ // check the replica set rather than the ISR.
+ if (Replicas.contains(partition.isr, brokerId)) {
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(brokerId).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ }
+
+ void removeLeaderships(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = chooseNewLeader(partition, isrWithoutCurLeader, false);
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(partition.isr)).
+ setLeader(newLeader).
+ setLeaderEpoch(partition.leaderEpoch + 1).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ ControllerResult<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request) {
+ boolean unclean = electionIsUnclean(request.electionType());
Review comment:
Hmm, I thought this already handles preferred leader election (there are only two options, PREFERRED and UNCLEAN, so far...)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] junrao commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
junrao commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573345382
##########
File path: metadata/src/main/java/org/apache/kafka/controller/Controller.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+
+public interface Controller extends AutoCloseable {
+ /**
+ * Change partition ISRs.
+ *
+ * @param request The AlterIsrRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request);
+
+ /**
+ * Create a batch of topics.
+ *
+ * @param request The CreateTopicsRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request);
+
+ /**
+ * Decommission a broker.
+ *
+ * @param brokerId The broker id to decommission.
+ *
+ * @return A future that is completed successfully hwne the broker is
+ * decommissioned, or if it is not registered in the first place.
+ */
+ CompletableFuture<Void> decommissionBroker(int brokerId);
+
+ /**
+ * Describe the current configuration of various resources.
+ *
+ * @param resources A map from resources to the collection of config keys that we
+ * want to describe for each. If the collection is empty, then
+ * all configuration keys will be described.
+ *
+ * @return
+ */
+ CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources);
+
+ /**
+ * Elect new partition leaders.
+ *
+ * @param request The request.
+ *
+ * @return A future yielding the elect leaders response.
+ */
+ CompletableFuture<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request);
+
+ /**
+ * Get the current finalized feature ranges for each feature.
+ *
+ * @return A future yielding the feature ranges.
+ */
+ CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures();
+
+ /**
+ * Perform some incremental configuration changes.
+ *
+ * @param configChanges The changes.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
Review comment:
The returned map is not keyed on partition.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/Controller.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+
+public interface Controller extends AutoCloseable {
+ /**
+ * Change partition ISRs.
+ *
+ * @param request The AlterIsrRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request);
+
+ /**
+ * Create a batch of topics.
+ *
+ * @param request The CreateTopicsRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request);
+
+ /**
+ * Decommission a broker.
+ *
+ * @param brokerId The broker id to decommission.
+ *
+ * @return A future that is completed successfully hwne the broker is
Review comment:
typo hwne
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,917 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
Review comment:
Could we move those private methods after all the internal classes?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
Review comment:
This can throw StaleBrokerEpochException. It would be useful for KafkaEventQueue.run() to log the event associated with the exception.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
Review comment:
We probably should name this sth like removeFromIsrAndMaybeChooseLeader.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(partitionData.partitionIndex()).
+ setTopicId(topic.id).
+ setIsr(partitionData.newIsr()).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+ return new ControllerResult<>(records, response);
+ }
+
+ void removeFromIsr(int brokerId, List<ApiMessageAndVersion> records) {
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, false);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] newIsr = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader, newLeaderEpoch;
+ if (partition.leader == brokerId) {
+ newLeader = partition.chooseNewLeader(newIsr, false);
+ newLeaderEpoch = partition.leaderEpoch + 1;
+ } else {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ }
+ records.add(new ApiMessageAndVersion(new IsrChangeRecord().
+ setPartitionId(topicPartition.partitionId()).
+ setTopicId(topic.id).
+ setIsr(Replicas.toList(newIsr)).
+ setLeader(newLeader).
+ setLeaderEpoch(newLeaderEpoch).
+ setPartitionEpoch(partition.partitionEpoch + 1), (short) 0));
+ }
+ }
+
+ List<ApiMessageAndVersion> removeLeaderships(int brokerId) {
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ Iterator<TopicPartition> iterator = brokersToIsrs.iterator(brokerId, true);
+ while (iterator.hasNext()) {
+ TopicPartition topicPartition = iterator.next();
+ TopicControlInfo topic = topics.get(topicPartition.topicId());
+ if (topic == null) {
+ throw new RuntimeException("Topic ID " + topicPartition.topicId() + " existed in " +
+ "isrMembers, but not in the topics map.");
+ }
+ PartitionControlInfo partition = topic.parts.get(topicPartition.partitionId());
+ if (partition == null) {
+ throw new RuntimeException("Partition " + topicPartition +
+ " existed in isrMembers, but not in the partitions map.");
+ }
+ int[] isrWithoutCurLeader = Replicas.copyWithout(partition.isr, brokerId);
+ int newLeader = partition.chooseNewLeader(isrWithoutCurLeader, false);
Review comment:
Hmm, not all partitions with ISR containing the shutting down need to change the leader.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,917 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
Review comment:
Do we also need to update the metric for processing time?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ControllerPurgatory.java
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Optional;
+import java.util.TreeMap;
+
+/**
+ * The purgatory which holds events that have been started, but not yet completed.
+ * We wait for the high water mark of the metadata log to advance before completing
+ * them.
+ */
+class ControllerPurgatory {
+ /**
+ * A map from log offsets to events. Each event will be completed once the log
+ * advances past its offset.
+ */
+ private final TreeMap<Long, List<DeferredEvent>> pending = new TreeMap<>();
+
+ /**
+ * Complete some purgatory entries.
+ *
+ * @param offset The offset which the high water mark has advanced to.
+ */
+ void completeUpTo(long offset) {
+ Iterator<Entry<Long, List<DeferredEvent>>> iter = pending.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<DeferredEvent>> entry = iter.next();
+ if (entry.getKey() > offset) {
+ break;
+ }
+ for (DeferredEvent event : entry.getValue()) {
+ event.complete(null);
+ }
+ iter.remove();
+ }
+ }
+
+ /**
+ * Fail all the pending purgatory entries.
+ *
+ * @param exception The exception to fail the entries with.
+ */
+ void failAll(Exception exception) {
+ Iterator<Entry<Long, List<DeferredEvent>>> iter = pending.entrySet().iterator();
+ while (iter.hasNext()) {
+ Entry<Long, List<DeferredEvent>> entry = iter.next();
+ for (DeferredEvent event : entry.getValue()) {
+ event.complete(exception);
+ }
+ iter.remove();
+ }
+ }
+
+ /**
+ * Add a new purgatory event.
+ *
+ * @param offset The offset to add the new event at.
+ * @param event The new event.
+ */
+ void add(long offset, DeferredEvent event) {
+ if (!pending.isEmpty()) {
+ long lastKey = pending.lastKey();
+ if (offset < lastKey) {
+ throw new RuntimeException("There is already a purgatory event with " +
+ "offset " + offset + ". We should not add one with an offset which " +
+ "is lower than or equal to that.");
Review comment:
Could we log lastKey too?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/Controller.java
##########
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+
+import java.util.Collection;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+
+public interface Controller extends AutoCloseable {
+ /**
+ * Change partition ISRs.
+ *
+ * @param request The AlterIsrRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request);
+
+ /**
+ * Create a batch of topics.
+ *
+ * @param request The CreateTopicsRequest data.
+ *
+ * @return A future yielding the response.
+ */
+ CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request);
+
+ /**
+ * Decommission a broker.
+ *
+ * @param brokerId The broker id to decommission.
+ *
+ * @return A future that is completed successfully hwne the broker is
+ * decommissioned, or if it is not registered in the first place.
+ */
+ CompletableFuture<Void> decommissionBroker(int brokerId);
+
+ /**
+ * Describe the current configuration of various resources.
+ *
+ * @param resources A map from resources to the collection of config keys that we
+ * want to describe for each. If the collection is empty, then
+ * all configuration keys will be described.
+ *
+ * @return
+ */
+ CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources);
+
+ /**
+ * Elect new partition leaders.
+ *
+ * @param request The request.
+ *
+ * @return A future yielding the elect leaders response.
+ */
+ CompletableFuture<ElectLeadersResponseData> electLeaders(ElectLeadersRequestData request);
+
+ /**
+ * Get the current finalized feature ranges for each feature.
+ *
+ * @return A future yielding the feature ranges.
+ */
+ CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures();
+
+ /**
+ * Perform some incremental configuration changes.
+ *
+ * @param configChanges The changes.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
+ */
+ CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Map.Entry<AlterConfigOp.OpType, String>>> configChanges,
+ boolean validateOnly);
+
+ /**
+ * Perform some configuration changes using the legacy API.
+ *
+ * @param newConfigs The new configuration maps to apply.
+ * @param validateOnly True if we should validate the changes but not apply them.
+ *
+ * @return A future yielding a map from partitions to error results.
Review comment:
The returned map is not keyed on partition.
##########
File path: metadata/src/main/java/org/apache/kafka/metalog/LocalLogManager.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.metalog;
+
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+
+/**
+ * The LocalLogManager is a test implementation that relies on the contents of memory.
+ */
+public final class LocalLogManager implements MetaLogManager, AutoCloseable {
+ interface LocalBatch {
+ int size();
+ }
+
+ static class LeaderChangeBatch implements LocalBatch {
+ private final MetaLogLeader newLeader;
+
+ LeaderChangeBatch(MetaLogLeader newLeader) {
+ this.newLeader = newLeader;
+ }
+
+ @Override
+ public int size() {
+ return 1;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof LeaderChangeBatch)) return false;
+ LeaderChangeBatch other = (LeaderChangeBatch) o;
+ if (!other.newLeader.equals(newLeader)) return false;
+ return true;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(newLeader);
+ }
+
+ @Override
+ public String toString() {
+ return "LeaderChangeBatch(newLeader=" + newLeader + ")";
+ }
+ }
+
+ static class LocalRecordBatch implements LocalBatch {
+ private final List<ApiMessage> records;
+
+ LocalRecordBatch(List<ApiMessage> records) {
+ this.records = records;
+ }
+
+ @Override
+ public int size() {
+ return records.size();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof LocalRecordBatch)) return false;
+ LocalRecordBatch other = (LocalRecordBatch) o;
+ if (!other.records.equals(records)) return false;
+ return true;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(records);
+ }
+
+ @Override
+ public String toString() {
+ return "LocalRecordBatch(records=" + records + ")";
+ }
+ }
+
+ public static class SharedLogData {
+ private final Logger log = LoggerFactory.getLogger(SharedLogData.class);
+ private final HashMap<Integer, LocalLogManager> logManagers = new HashMap<>();
+ private final TreeMap<Long, LocalBatch> batches = new TreeMap<>();
Review comment:
Could we add a bit comment explaining logManagers and batches?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,917 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
Review comment:
The return type is not a tuple.
##########
File path: metadata/src/main/java/org/apache/kafka/metalog/LocalLogManager.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.metalog;
+
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+
+/**
+ * The LocalLogManager is a test implementation that relies on the contents of memory.
+ */
+public final class LocalLogManager implements MetaLogManager, AutoCloseable {
+ interface LocalBatch {
+ int size();
+ }
+
+ static class LeaderChangeBatch implements LocalBatch {
+ private final MetaLogLeader newLeader;
+
+ LeaderChangeBatch(MetaLogLeader newLeader) {
+ this.newLeader = newLeader;
+ }
+
+ @Override
+ public int size() {
+ return 1;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof LeaderChangeBatch)) return false;
+ LeaderChangeBatch other = (LeaderChangeBatch) o;
+ if (!other.newLeader.equals(newLeader)) return false;
+ return true;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(newLeader);
+ }
+
+ @Override
+ public String toString() {
+ return "LeaderChangeBatch(newLeader=" + newLeader + ")";
+ }
+ }
+
+ static class LocalRecordBatch implements LocalBatch {
+ private final List<ApiMessage> records;
+
+ LocalRecordBatch(List<ApiMessage> records) {
+ this.records = records;
+ }
+
+ @Override
+ public int size() {
+ return records.size();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof LocalRecordBatch)) return false;
+ LocalRecordBatch other = (LocalRecordBatch) o;
+ if (!other.records.equals(records)) return false;
+ return true;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(records);
+ }
+
+ @Override
+ public String toString() {
+ return "LocalRecordBatch(records=" + records + ")";
+ }
+ }
+
+ public static class SharedLogData {
+ private final Logger log = LoggerFactory.getLogger(SharedLogData.class);
+ private final HashMap<Integer, LocalLogManager> logManagers = new HashMap<>();
+ private final TreeMap<Long, LocalBatch> batches = new TreeMap<>();
+ private MetaLogLeader leader = new MetaLogLeader(-1, -1);
+ private long prevOffset = -1;
+
+ synchronized void registerLogManager(LocalLogManager logManager) {
+ if (logManagers.put(logManager.nodeId(), logManager) != null) {
+ throw new RuntimeException("Can't have multiple LocalLogManagers " +
+ "with id " + logManager.nodeId());
+ }
+ electLeaderIfNeeded();
+ }
+
+ synchronized void unregisterLogManager(LocalLogManager logManager) {
+ if (!logManagers.remove(logManager.nodeId(), logManager)) {
+ throw new RuntimeException("Log manager " + logManager.nodeId() +
+ " was not found.");
+ }
+ }
+
+ synchronized long tryAppend(int nodeId, long epoch, LocalBatch batch) {
+ if (epoch != leader.epoch()) {
+ log.trace("tryAppend(nodeId={}, epoch={}): the provided epoch does not " +
+ "match the current leader epoch of {}.", nodeId, epoch, leader.epoch());
+ return Long.MAX_VALUE;
+ }
+ if (nodeId != leader.nodeId()) {
+ log.trace("tryAppend(nodeId={}, epoch={}): the given node id does not " +
+ "match the current leader id of {}.", nodeId, leader.nodeId());
+ return Long.MAX_VALUE;
+ }
+ log.trace("tryAppend(nodeId={}): appending {}.", nodeId, batch);
+ long offset = append(batch);
+ electLeaderIfNeeded();
+ return offset;
+ }
+
+ synchronized long append(LocalBatch batch) {
+ prevOffset += batch.size();
+ log.debug("append(batch={}, prevOffset={})", batch, prevOffset);
+ batches.put(prevOffset, batch);
+ if (batch instanceof LeaderChangeBatch) {
+ LeaderChangeBatch leaderChangeBatch = (LeaderChangeBatch) batch;
+ leader = leaderChangeBatch.newLeader;
+ }
+ for (LocalLogManager logManager : logManagers.values()) {
+ logManager.scheduleLogCheck();
+ }
+ return prevOffset;
+ }
+
+ synchronized void electLeaderIfNeeded() {
+ if (leader.nodeId() != -1 || logManagers.isEmpty()) {
+ return;
+ }
+ int nextLeaderIndex = ThreadLocalRandom.current().nextInt(logManagers.size());
+ Iterator<Integer> iter = logManagers.keySet().iterator();
+ Integer nextLeaderNode = null;
+ for (int i = 0; i <= nextLeaderIndex; i++) {
+ nextLeaderNode = iter.next();
+ }
+ MetaLogLeader newLeader = new MetaLogLeader(nextLeaderNode, leader.epoch() + 1);
+ log.info("Elected new leader: {}.", newLeader);
+ append(new LeaderChangeBatch(newLeader));
+ }
+
+ synchronized Entry<Long, LocalBatch> nextBatch(long offset) {
+ Entry<Long, LocalBatch> entry = batches.higherEntry(offset);
+ if (entry == null) {
+ return null;
+ }
+ return new SimpleImmutableEntry<>(entry.getKey(), entry.getValue());
+ }
+ }
+
+ private static class MetaLogListenerData {
+ private long offset = -1;
+ private final MetaLogListener listener;
+
+ MetaLogListenerData(MetaLogListener listener) {
+ this.listener = listener;
+ }
+ }
+
+ private final Logger log;
+
+ private final int nodeId;
+
+ private final SharedLogData shared;
+
+ private final EventQueue eventQueue;
+
+ private boolean initialized = false;
+
+ private boolean shutdown = false;
+
+ private long maxReadOffset = Long.MAX_VALUE;
Review comment:
Could we add a comment for maxReadOffset? Is it the committed offset?
##########
File path: metadata/src/main/java/org/apache/kafka/metalog/LocalLogManager.java
##########
@@ -0,0 +1,379 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.metalog;
+
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.TreeMap;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.stream.Collectors;
+
+
+/**
+ * The LocalLogManager is a test implementation that relies on the contents of memory.
Review comment:
Is this used only for test?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
Review comment:
This is in response to a heartBeat request. So, it should generate a response in ControllerResult?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
Review comment:
We already replay the message when it's first appended to the log and here we replay the same message again after commit. This could temporarily revert the state. For example, the latest (uncommitted) config could be overwritten by a previously committed config.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
Review comment:
We need to choose at least a live replica.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
Review comment:
In the ZK based code, we also take live brokers into consideration when selecting a new leader.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java
##########
@@ -0,0 +1,367 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef.ConfigKey;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource.Type;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.APPEND;
+
+public class ConfigurationControlManager {
+ private final Logger log;
+ private final SnapshotRegistry snapshotRegistry;
+ private final Map<ConfigResource.Type, ConfigDef> configDefs;
+ private final TimelineHashMap<ConfigResource, TimelineHashMap<String, String>> configData;
+
+ ConfigurationControlManager(LogContext logContext,
+ SnapshotRegistry snapshotRegistry,
+ Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.log = logContext.logger(ConfigurationControlManager.class);
+ this.snapshotRegistry = snapshotRegistry;
+ this.configDefs = configDefs;
+ this.configData = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+
+ /**
+ * Determine the result of applying a batch of incremental configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * Note that there can only be one result per ConfigResource. So if you try to modify
+ * several keys and one modification fails, the whole ConfigKey fails and nothing gets
+ * changed.
+ *
+ * @param configChanges Maps each resource to a map from config keys to
+ * operation data.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, Entry<OpType, String>>> resourceEntry :
+ configChanges.entrySet()) {
+ incrementalAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void incrementalAlterConfigResource(ConfigResource configResource,
+ Map<String, Entry<OpType, String>> keysToOps,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ for (Entry<String, Entry<OpType, String>> keysToOpsEntry : keysToOps.entrySet()) {
+ String key = keysToOpsEntry.getKey();
+ String currentValue = null;
+ TimelineHashMap<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs != null) {
+ currentValue = currentConfigs.get(key);
+ }
+ String newValue = currentValue;
+ Entry<OpType, String> opTypeAndNewValue = keysToOpsEntry.getValue();
+ OpType opType = opTypeAndNewValue.getKey();
+ String opValue = opTypeAndNewValue.getValue();
+ switch (opType) {
+ case SET:
+ newValue = opValue;
+ break;
+ case DELETE:
+ if (opValue != null) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_REQUEST, "A DELETE op was given with a " +
+ "non-null value."));
+ return;
+ }
+ newValue = null;
+ break;
+ case APPEND:
+ case SUBTRACT:
+ if (!isSplittable(configResource.type(), key)) {
+ outputResults.put(configResource, new ApiError(
+ Errors.INVALID_CONFIG, "Can't " + opType + " to " +
+ "key " + key + " because its type is not LIST."));
+ return;
+ }
+ List<String> newValueParts = getParts(newValue, key, configResource);
+ if (opType == APPEND) {
+ if (!newValueParts.contains(opValue)) {
+ newValueParts.add(opValue);
+ }
+ newValue = String.join(",", newValueParts);
+ } else if (newValueParts.remove(opValue)) {
+ newValue = String.join(",", newValueParts);
+ }
+ break;
+ }
+ if (!Objects.equals(currentValue, newValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ /**
+ * Determine the result of applying a batch of legacy configuration changes. Note
+ * that this method does not change the contents of memory. It just generates a
+ * result, that you can replay later if you wish using replay().
+ *
+ * @param newConfigs The new configurations to install for each resource.
+ * All existing configurations will be overwritten.
+ * @return The result.
+ */
+ ControllerResult<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs) {
+ List<ApiMessageAndVersion> outputRecords = new ArrayList<>();
+ Map<ConfigResource, ApiError> outputResults = new HashMap<>();
+ for (Entry<ConfigResource, Map<String, String>> resourceEntry :
+ newConfigs.entrySet()) {
+ legacyAlterConfigResource(resourceEntry.getKey(),
+ resourceEntry.getValue(),
+ outputRecords,
+ outputResults);
+ }
+ return new ControllerResult<>(outputRecords, outputResults);
+ }
+
+ private void legacyAlterConfigResource(ConfigResource configResource,
+ Map<String, String> newConfigs,
+ List<ApiMessageAndVersion> outputRecords,
+ Map<ConfigResource, ApiError> outputResults) {
+ ApiError error = checkConfigResource(configResource);
+ if (error.isFailure()) {
+ outputResults.put(configResource, error);
+ return;
+ }
+ List<ApiMessageAndVersion> newRecords = new ArrayList<>();
+ Map<String, String> currentConfigs = configData.get(configResource);
+ if (currentConfigs == null) {
+ currentConfigs = Collections.emptyMap();
+ }
+ for (Entry<String, String> entry : newConfigs.entrySet()) {
+ String key = entry.getKey();
+ String newValue = entry.getValue();
+ String currentValue = currentConfigs.get(key);
+ if (!Objects.equals(newValue, currentValue)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(newValue), (short) 0));
+ }
+ }
+ for (String key : currentConfigs.keySet()) {
+ if (!newConfigs.containsKey(key)) {
+ newRecords.add(new ApiMessageAndVersion(new ConfigRecord().
+ setResourceType(configResource.type().id()).
+ setResourceName(configResource.name()).
+ setName(key).
+ setValue(null), (short) 0));
+ }
+ }
+ outputRecords.addAll(newRecords);
+ outputResults.put(configResource, ApiError.NONE);
+ }
+
+ private List<String> getParts(String value, String key, ConfigResource configResource) {
+ if (value == null) {
+ value = getConfigValueDefault(configResource.type(), key);
+ }
+ List<String> parts = new ArrayList<>();
+ if (value == null) {
+ return parts;
+ }
+ String[] splitValues = value.split(",");
+ for (String splitValue : splitValues) {
+ if (!splitValue.isEmpty()) {
+ parts.add(splitValue);
+ }
+ }
+ return parts;
+ }
+
+ static ApiError checkConfigResource(ConfigResource configResource) {
+ switch (configResource.type()) {
+ case BROKER_LOGGER:
+ // We do not handle resources of type BROKER_LOGGER in
Review comment:
Is that temporary?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
Review comment:
We already did this check and the one below in the caller through `clusterControl.checkBrokerEpoch`.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
Review comment:
In the ZK case, we use the ZK version to do conditional updates. In Raft, could we associated each partitionState with the offset in the Raft log and use that as partitionEpoch for conditional updates? This way, we don't need to explicitly maintain a separate partitionEpoch field and the epoch is automatically bumped up for any change to the partition record, not just for leader and isr.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
Review comment:
This seems unnecessary.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,712 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+public class ReplicationControlManager {
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(IsrChangeRecord record) {
+ return new PartitionControlInfo(replicas,
+ Replicas.toArray(record.isr()),
+ removingReplicas,
+ addingReplicas,
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("isr=").append(Arrays.toString(isr));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("removingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ prefix = ", ";
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("addingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ prefix = ", ";
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("leader=").append(leader);
+ prefix = ", ";
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("leaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("partitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+
+ int chooseNewLeader(int[] newIsr, boolean unclean) {
+ for (int i = 0; i < replicas.length; i++) {
+ int replica = replicas[i];
+ if (Replicas.contains(newIsr, replica)) {
+ return replica;
+ }
+ }
+ if (unclean && replicas.length > 0) {
+ return replicas[0];
+ }
+ return -1;
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, -1, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(IsrChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ }
+ int newLeader, newLeaderEpoch;
+ if (Replicas.contains(newIsr, partition.leader)) {
+ newLeader = partition.leader;
+ newLeaderEpoch = partition.leaderEpoch;
+ } else {
+ newLeader = partition.chooseNewLeader(newIsr, false);
Review comment:
Currently, the follower never removes the leader out of ISR. So, perhaps we should just throw an exception if this is not the case.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
Review comment:
Some of the replay (e.g. UNREGISTER_BROKER_RECORD) could throw exceptions. We probably need to turn the exception into an error response. Are we handling that already?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
##########
@@ -0,0 +1,456 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.Endpoint;
+import org.apache.kafka.common.errors.DuplicateBrokerRegistrationException;
+import org.apache.kafka.common.errors.StaleBrokerEpochException;
+import org.apache.kafka.common.errors.UnsupportedVersionException;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.security.auth.SecurityProtocol;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+
+public class ClusterControlManager {
+ class ReadyBrokersFuture {
+ private final CompletableFuture<Void> future;
+ private final int minBrokers;
+
+ ReadyBrokersFuture(CompletableFuture<Void> future, int minBrokers) {
+ this.future = future;
+ this.minBrokers = minBrokers;
+ }
+
+ boolean check() {
+ int numUnfenced = 0;
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ if (!registration.fenced()) {
+ numUnfenced++;
+ }
+ if (numUnfenced >= minBrokers) {
+ return true;
+ }
+ }
+ return false;
+ }
+ }
+
+ /**
+ * The SLF4J log context.
+ */
+ private final LogContext logContext;
+
+ /**
+ * The SLF4J log object.
+ */
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * How long sessions should last, in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * The replica placement policy to use.
+ */
+ private final ReplicaPlacementPolicy placementPolicy;
+
+ /**
+ * Maps broker IDs to broker registrations.
+ */
+ private final TimelineHashMap<Integer, BrokerRegistration> brokerRegistrations;
+
+ /**
+ * The broker heartbeat manager, or null if this controller is on standby.
+ */
+ private BrokerHeartbeatManager heartbeatManager;
+
+ /**
+ * A future which is completed as soon as we have the given number of brokers
+ * ready.
+ */
+ private Optional<ReadyBrokersFuture> readyBrokersFuture;
+
+ ClusterControlManager(LogContext logContext,
+ Time time,
+ SnapshotRegistry snapshotRegistry,
+ long sessionTimeoutNs,
+ ReplicaPlacementPolicy placementPolicy) {
+ this.logContext = logContext;
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.placementPolicy = placementPolicy;
+ this.brokerRegistrations = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.heartbeatManager = null;
+ this.readyBrokersFuture = Optional.empty();
+ }
+
+ /**
+ * Transition this ClusterControlManager to active.
+ */
+ public void activate() {
+ heartbeatManager = new BrokerHeartbeatManager(logContext, time, sessionTimeoutNs);
+ for (BrokerRegistration registration : brokerRegistrations.values()) {
+ heartbeatManager.touch(registration.id(), registration.fenced(), -1);
+ }
+ }
+
+ /**
+ * Transition this ClusterControlManager to standby.
+ */
+ public void deactivate() {
+ heartbeatManager = null;
+ }
+
+ // VisibleForTesting
+ Map<Integer, BrokerRegistration> brokerRegistrations() {
+ return brokerRegistrations;
+ }
+
+ /**
+ * Process an incoming broker registration request.
+ */
+ public ControllerResult<BrokerRegistrationReply> registerBroker(
+ BrokerRegistrationRequestData request, long brokerEpoch,
+ FeatureManager.FinalizedFeaturesAndEpoch finalizedFeaturesAndEpoch) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing != null) {
+ if (heartbeatManager.hasValidSession(brokerId)) {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ throw new DuplicateBrokerRegistrationException("Another broker is " +
+ "registered with that broker id.");
+ }
+ } else {
+ if (!existing.incarnationId().equals(request.incarnationId())) {
+ // Remove any existing session for the old broker incarnation.
+ heartbeatManager.remove(brokerId);
+ existing = null;
+ }
+ }
+ }
+
+ RegisterBrokerRecord record = new RegisterBrokerRecord().setBrokerId(brokerId).
+ setIncarnationId(request.incarnationId()).
+ setBrokerEpoch(brokerEpoch).
+ setRack(request.rack());
+ for (BrokerRegistrationRequestData.Listener listener : request.listeners()) {
+ record.endPoints().add(new RegisterBrokerRecord.BrokerEndpoint().
+ setHost(listener.host()).
+ setName(listener.name()).
+ setPort(listener.port()).
+ setSecurityProtocol(listener.securityProtocol()));
+ }
+ for (BrokerRegistrationRequestData.Feature feature : request.features()) {
+ VersionRange supported = finalizedFeaturesAndEpoch.finalizedFeatures().
+ getOrDefault(feature.name(), VersionRange.ALL);
+ if (!supported.contains(new VersionRange(feature.minSupportedVersion(),
+ feature.maxSupportedVersion()))) {
+ throw new UnsupportedVersionException("Unable to register because " +
+ "the broker has an unsupported version of " + feature.name());
+ }
+ record.features().add(new RegisterBrokerRecord.BrokerFeature().
+ setName(feature.name()).
+ setMinSupportedVersion(feature.minSupportedVersion()).
+ setMaxSupportedVersion(feature.maxSupportedVersion()));
+ }
+
+ if (existing == null) {
+ heartbeatManager.touch(brokerId, true, -1);
+ } else {
+ heartbeatManager.touch(brokerId, existing.fenced(), -1);
+ }
+
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(record, (short) 0));
+ return new ControllerResult<>(records, new BrokerRegistrationReply(brokerEpoch));
+ }
+
+ public ControllerResult<Void> decommissionBroker(int brokerId) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ BrokerRegistration existing = brokerRegistrations.get(brokerId);
+ if (existing == null) {
+ return new ControllerResult<>(new ArrayList<>(), null);
+ }
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ records.add(new ApiMessageAndVersion(
+ new UnregisterBrokerRecord().
+ setBrokerId(brokerId).
+ setBrokerEpoch(existing.epoch()),
+ (short) 0));
+ heartbeatManager.remove(brokerId);
+ return new ControllerResult<>(records, null);
+ }
+
+ public ControllerResult<BrokerHeartbeatReply> processBrokerHeartbeat(
+ BrokerHeartbeatRequestData request,
+ long lastCommittedOffset,
+ boolean movingLeadersForShutDown) {
+ if (heartbeatManager == null) {
+ throw new RuntimeException("ClusterControlManager is not active.");
+ }
+ int brokerId = request.brokerId();
+ BrokerRegistration registration = brokerRegistrations.get(brokerId);
+ if (registration == null) {
+ throw new StaleBrokerEpochException("No registration found for broker " +
+ request.brokerId());
+ }
+ if (request.brokerEpoch() != registration.epoch()) {
+ throw new StaleBrokerEpochException("Expected broker epoch " +
+ request.brokerEpoch() + "; got epoch " + registration.epoch());
+ }
+ boolean isCaughtUp = request.currentMetadataOffset() >= lastCommittedOffset;
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean isFenced = registration.fenced();
+ boolean shouldShutdown = false;
+ if (isFenced) {
+ if (request.wantShutDown()) {
+ // If the broker is fenced, and requests a shutdown, do it immediately.
+ isFenced = true;
+ shouldShutdown = true;
+ } else if (isCaughtUp && !request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new UnfenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = false;
+ shouldShutdown = false;
+ }
+ } else {
+ if (request.wantFence()) {
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
+ setId(brokerId).setEpoch(request.brokerEpoch()), (short) 0));
+ isFenced = true;
+ shouldShutdown = request.wantShutDown();
+ } else {
+ if (request.wantShutDown()) {
+ heartbeatManager.beginBrokerShutDown(request.brokerId(), movingLeadersForShutDown);
+ }
+ if (heartbeatManager.shouldShutDown(request.brokerId())) {
+ isFenced = true;
+ shouldShutdown = true;
+ records.add(new ApiMessageAndVersion(new FenceBrokerRecord().
Review comment:
If we want to log the shutting down the broker, it seems it's more consistent if we always log it. Now, it seems we log it only when leaders need to be moved.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
Review comment:
I thought the raft leader epoch is an int since we store only int as leader epoch in the log?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
##########
@@ -0,0 +1,920 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.ApiException;
+import org.apache.kafka.common.errors.NotControllerException;
+import org.apache.kafka.common.errors.UnknownServerException;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.BrokerRegistrationRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.ConfigRecord;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.IsrChangeRecord;
+import org.apache.kafka.common.metadata.MetadataRecordType;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.QuotaRecord;
+import org.apache.kafka.common.metadata.RegisterBrokerRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.ApiMessage;
+import org.apache.kafka.common.quota.ClientQuotaAlteration;
+import org.apache.kafka.common.quota.ClientQuotaEntity;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.common.utils.Utils;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistrationReply;
+import org.apache.kafka.metadata.FeatureManager;
+import org.apache.kafka.metadata.VersionRange;
+import org.apache.kafka.metalog.MetaLogLeader;
+import org.apache.kafka.metalog.MetaLogListener;
+import org.apache.kafka.metalog.MetaLogManager;
+import org.apache.kafka.queue.EventQueue.EarliestDeadlineFunction;
+import org.apache.kafka.queue.EventQueue;
+import org.apache.kafka.queue.KafkaEventQueue;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.slf4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map.Entry;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+
+
+public final class QuorumController implements Controller {
+ /**
+ * A builder class which creates the QuorumController.
+ */
+ static public class Builder {
+ private final int nodeId;
+ private Time time = Time.SYSTEM;
+ private String threadNamePrefix = null;
+ private LogContext logContext = null;
+ private Map<ConfigResource.Type, ConfigDef> configDefs = Collections.emptyMap();
+ private MetaLogManager logManager = null;
+ private Map<String, VersionRange> supportedFeatures = Collections.emptyMap();
+ private short defaultReplicationFactor = 3;
+ private int defaultNumPartitions = 1;
+ private ReplicaPlacementPolicy replicaPlacementPolicy =
+ new SimpleReplicaPlacementPolicy(new Random());
+ private long sessionTimeoutNs = TimeUnit.NANOSECONDS.convert(18, TimeUnit.SECONDS);
+
+ public Builder(int nodeId) {
+ this.nodeId = nodeId;
+ }
+
+ public Builder setTime(Time time) {
+ this.time = time;
+ return this;
+ }
+
+ public Builder setThreadNamePrefix(String threadNamePrefix) {
+ this.threadNamePrefix = threadNamePrefix;
+ return this;
+ }
+
+ public Builder setLogContext(LogContext logContext) {
+ this.logContext = logContext;
+ return this;
+ }
+
+ public Builder setConfigDefs(Map<ConfigResource.Type, ConfigDef> configDefs) {
+ this.configDefs = configDefs;
+ return this;
+ }
+
+ public Builder setLogManager(MetaLogManager logManager) {
+ this.logManager = logManager;
+ return this;
+ }
+
+ public Builder setSupportedFeatures(Map<String, VersionRange> supportedFeatures) {
+ this.supportedFeatures = supportedFeatures;
+ return this;
+ }
+
+ public Builder setDefaultReplicationFactor(short defaultReplicationFactor) {
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ return this;
+ }
+
+ public Builder setDefaultNumPartitions(int defaultNumPartitions) {
+ this.defaultNumPartitions = defaultNumPartitions;
+ return this;
+ }
+
+ public Builder setReplicaPlacementPolicy(ReplicaPlacementPolicy replicaPlacementPolicy) {
+ this.replicaPlacementPolicy = replicaPlacementPolicy;
+ return this;
+ }
+
+ public Builder setSessionTimeoutNs(long sessionTimeoutNs) {
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ return this;
+ }
+
+ public QuorumController build() throws Exception {
+ if (logManager == null) {
+ throw new RuntimeException("You must set a metadata log manager.");
+ }
+ if (threadNamePrefix == null) {
+ threadNamePrefix = String.format("Node%d_", nodeId);
+ }
+ if (logContext == null) {
+ logContext = new LogContext(String.format("[Controller %d] ", nodeId));
+ }
+ KafkaEventQueue queue = null;
+ try {
+ queue = new KafkaEventQueue(time, logContext, threadNamePrefix);
+ return new QuorumController(logContext, nodeId, queue, time, configDefs,
+ logManager, supportedFeatures, defaultReplicationFactor,
+ defaultNumPartitions, replicaPlacementPolicy, sessionTimeoutNs);
+ } catch (Exception e) {
+ Utils.closeQuietly(queue, "event queue");
+ throw e;
+ }
+ }
+ }
+
+ private static final String ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX =
+ "The active controller appears to be node ";
+
+ private NotControllerException newNotControllerException() {
+ int latestController = logManager.leader().nodeId();
+ if (latestController < 0) {
+ return new NotControllerException("No controller appears to be active.");
+ } else {
+ return new NotControllerException(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX +
+ latestController);
+ }
+ }
+
+ public static int exceptionToApparentController(NotControllerException e) {
+ if (e.getMessage().startsWith(ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX)) {
+ return Integer.parseInt(e.getMessage().substring(
+ ACTIVE_CONTROLLER_EXCEPTION_TEXT_PREFIX.length()));
+ } else {
+ return -1;
+ }
+ }
+
+ private void handleEventEnd(String name, long startProcessingTimeNs) {
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs;
+ log.debug("Processed {} in {} us", name,
+ TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS));
+ }
+
+ private Throwable handleEventException(String name,
+ Optional<Long> startProcessingTimeNs,
+ Throwable exception) {
+ if (!startProcessingTimeNs.isPresent()) {
+ log.debug("unable to start processing {} because of {}.", name,
+ exception.getClass().getSimpleName());
+ if (exception instanceof ApiException) {
+ return exception;
+ } else {
+ return new UnknownServerException(exception);
+ }
+ }
+ long endProcessingTime = time.nanoseconds();
+ long deltaNs = endProcessingTime - startProcessingTimeNs.get();
+ long deltaUs = TimeUnit.MICROSECONDS.convert(deltaNs, TimeUnit.NANOSECONDS);
+ if (exception instanceof ApiException) {
+ log.debug("{}: failed with {} in {} us", name,
+ exception.getClass().getSimpleName(), deltaUs);
+ return exception;
+ }
+ log.warn("{}: failed with unknown server exception {} at epoch {} in {} us. " +
+ "Reverting to last committed offset {}.",
+ this, exception.getClass().getSimpleName(), curClaimEpoch, deltaUs,
+ lastCommittedOffset, exception);
+ renounce();
+ return new UnknownServerException(exception);
+ }
+
+ /**
+ * A controller event for handling internal state changes, such as Raft inputs.
+ */
+ class ControlEvent implements EventQueue.Event {
+ private final String name;
+ private final Runnable handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControlEvent(String name, Runnable handler) {
+ this.name = name;
+ this.handler = handler;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ log.debug("Executing {}.", this);
+ handler.run();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ handleEventException(name, startProcessingTimeNs, exception);
+ }
+
+ @Override
+ public String toString() {
+ return name;
+ }
+ }
+
+ private void appendControlEvent(String name, Runnable handler) {
+ ControlEvent event = new ControlEvent(name, handler);
+ queue.append(event);
+ }
+
+ /**
+ * A controller event that reads the committed internal state in order to expose it
+ * to an API.
+ */
+ class ControllerReadEvent<T> implements EventQueue.Event {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final Supplier<T> handler;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+
+ ControllerReadEvent(String name, Supplier<T> handler) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.handler = handler;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ T value = handler.get();
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(value);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ // VisibleForTesting
+ ReplicationControlManager replicationControl() {
+ return replicationControl;
+ }
+
+ // VisibleForTesting
+ <T> CompletableFuture<T> appendReadEvent(String name, Supplier<T> handler) {
+ ControllerReadEvent<T> event = new ControllerReadEvent<T>(name, handler);
+ queue.append(event);
+ return event.future();
+ }
+
+ interface ControllerWriteOperation<T> {
+ /**
+ * Generate the metadata records needed to implement this controller write
+ * operation. In general, this operation should not modify the "hard state" of
+ * the controller. That modification will happen later on, when we replay the
+ * records generated by this function.
+ *
+ * There are cases where this function modifies the "soft state" of the
+ * controller. Mainly, this happens when we process cluster heartbeats.
+ *
+ * This function also generates an RPC result. In general, if the RPC resulted in
+ * an error, the RPC result will be an error, and the generated record list will
+ * be empty. This would happen if we tried to create a topic with incorrect
+ * parameters, for example. Of course, partial errors are possible for batch
+ * operations.
+ *
+ * @return A tuple containing a list of records, and an RPC result.
+ */
+ ControllerResult<T> generateRecordsAndResult() throws Exception;
+
+ /**
+ * Once we've passed the records to the Raft layer, we will invoke this function
+ * with the end offset at which those records were placed. If there were no
+ * records to write, we'll just pass the last write offset.
+ */
+ default void processBatchEndOffset(long offset) {}
+ }
+
+ /**
+ * A controller event that modifies the controller state.
+ */
+ class ControllerWriteEvent<T> implements EventQueue.Event, DeferredEvent {
+ private final String name;
+ private final CompletableFuture<T> future;
+ private final ControllerWriteOperation<T> op;
+ private Optional<Long> startProcessingTimeNs = Optional.empty();
+ private ControllerResultAndOffset<T> resultAndOffset;
+
+ ControllerWriteEvent(String name, ControllerWriteOperation<T> op) {
+ this.name = name;
+ this.future = new CompletableFuture<T>();
+ this.op = op;
+ this.resultAndOffset = null;
+ }
+
+ CompletableFuture<T> future() {
+ return future;
+ }
+
+ @Override
+ public void run() throws Exception {
+ long controllerEpoch = curClaimEpoch;
+ if (controllerEpoch == -1) {
+ throw newNotControllerException();
+ }
+ startProcessingTimeNs = Optional.of(time.nanoseconds());
+ ControllerResult<T> result = op.generateRecordsAndResult();
+ if (result.records().isEmpty()) {
+ op.processBatchEndOffset(writeOffset);
+ // If the operation did not return any records, then it was actually just
+ //a read after all, and not a read + write. However, this read was done
+ //from the latest in-memory state, which might contain uncommitted data.
+ Optional<Long> maybeOffset = purgatory.highestPendingOffset();
+ if (!maybeOffset.isPresent()) {
+ // If the purgatory is empty, there are no pending operations and no
+ // uncommitted state. We can return immediately.
+ this.resultAndOffset = new ControllerResultAndOffset<>(-1,
+ new ArrayList<>(), result.response());
+ log.debug("Completing read-only operation {} immediately because " +
+ "the purgatory is empty.", this);
+ complete(null);
+ return;
+ }
+ // If there are operations in the purgatory, we want to wait for the latest
+ // one to complete before returning our result to the user.
+ this.resultAndOffset = new ControllerResultAndOffset<>(maybeOffset.get(),
+ result.records(), result.response());
+ log.debug("Read-only operation {} will be completed when the log " +
+ "reaches offset {}", this, resultAndOffset.offset());
+ } else {
+ // If the operation returned a batch of records, those records need to be
+ // written before we can return our result to the user. Here, we hand off
+ // the batch of records to the metadata log manager. They will be written
+ // out asynchronously.
+ long offset = logManager.scheduleWrite(controllerEpoch, result.records());
+ op.processBatchEndOffset(offset);
+ writeOffset = offset;
+ this.resultAndOffset = new ControllerResultAndOffset<>(offset,
+ result.records(), result.response());
+ for (ApiMessageAndVersion message : result.records()) {
+ replay(message.message());
+ }
+ snapshotRegistry.createSnapshot(offset);
+ log.debug("Read-write operation {} will be completed when the log " +
+ "reaches offset {}.", this, resultAndOffset.offset());
+ }
+ purgatory.add(resultAndOffset.offset(), this);
+ }
+
+ @Override
+ public void handleException(Throwable exception) {
+ complete(exception);
+ }
+
+ @Override
+ public void complete(Throwable exception) {
+ if (exception == null) {
+ handleEventEnd(this.toString(), startProcessingTimeNs.get());
+ future.complete(resultAndOffset.response());
+ } else {
+ future.completeExceptionally(
+ handleEventException(name, startProcessingTimeNs, exception));
+ }
+ }
+
+ @Override
+ public String toString() {
+ return name + "(" + System.identityHashCode(this) + ")";
+ }
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ long timeoutMs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.appendWithDeadline(time.nanoseconds() +
+ TimeUnit.NANOSECONDS.convert(timeoutMs, TimeUnit.MILLISECONDS), event);
+ return event.future();
+ }
+
+ private <T> CompletableFuture<T> appendWriteEvent(String name,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.append(event);
+ return event.future();
+ }
+
+ class QuorumMetaLogListener implements MetaLogListener {
+ @Override
+ public void handleCommits(long offset, List<ApiMessage> messages) {
+ appendControlEvent("handleCommits[" + offset + "]", () -> {
+ if (curClaimEpoch == -1) {
+ if (log.isDebugEnabled()) {
+ if (log.isTraceEnabled()) {
+ log.trace("Replaying commits from the active node up to " +
+ "offset {}: {}.", offset, messages.stream().
+ map(m -> m.toString()).collect(Collectors.joining(", ")));
+ } else {
+ log.debug("Replaying commits from the active node up to " +
+ "offset {}.", offset);
+ }
+ }
+ for (ApiMessage message : messages) {
+ replay(message);
+ }
+ } else {
+ log.debug("Completing purgatory items up to offset {}.", offset);
+
+ // Complete any events in the purgatory that were waiting for this offset.
+ purgatory.completeUpTo(offset);
+
+ // Delete all snapshots older than the offset.
+ // TODO: add an exception here for when we're writing out a log snapshot
+ snapshotRegistry.deleteSnapshotsUpTo(offset);
+ }
+ lastCommittedOffset = offset;
+ });
+ }
+
+ @Override
+ public void handleNewLeader(MetaLogLeader newLeader) {
+ if (newLeader.nodeId() == nodeId) {
+ final long newEpoch = newLeader.epoch();
+ appendControlEvent("handleClaim[" + newEpoch + "]", () -> {
+ long curEpoch = curClaimEpoch;
+ if (curEpoch != -1) {
+ throw new RuntimeException("Tried to claim controller epoch " +
+ newEpoch + ", but we never renounced controller epoch " +
+ curEpoch);
+ }
+ log.info("Becoming active at controller epoch {}.", newEpoch);
+ curClaimEpoch = newEpoch;
+ writeOffset = lastCommittedOffset;
+ clusterControl.activate();
+ });
+ }
+ }
+
+ @Override
+ public void handleRenounce(long oldEpoch) {
+ appendControlEvent("handleRenounce[" + oldEpoch + "]", () -> {
+ if (curClaimEpoch == oldEpoch) {
+ log.info("Renouncing the leadership at oldEpoch {} due to a metadata " +
+ "log event. Reverting to last committed offset {}.", curClaimEpoch,
+ lastCommittedOffset);
+ renounce();
+ }
+ });
+ }
+
+ @Override
+ public void beginShutdown() {
+ queue.beginShutdown("MetaLogManager.Listener");
+ }
+ }
+
+ private void renounce() {
+ curClaimEpoch = -1;
+ purgatory.failAll(newNotControllerException());
+ snapshotRegistry.revertToSnapshot(lastCommittedOffset);
+ snapshotRegistry.deleteSnapshotsUpTo(lastCommittedOffset);
+ writeOffset = -1;
+ clusterControl.deactivate();
+ cancelMaybeFenceReplicas();
+ }
+
+ private <T> void scheduleDeferredWriteEvent(String name, long deadlineNs,
+ ControllerWriteOperation<T> op) {
+ ControllerWriteEvent<T> event = new ControllerWriteEvent<>(name, op);
+ queue.scheduleDeferred(name, new EarliestDeadlineFunction(deadlineNs), event);
+ event.future.exceptionally(e -> {
+ if (e instanceof UnknownServerException && e.getCause() != null &&
+ e.getCause() instanceof RejectedExecutionException) {
+ log.error("Cancelling write event {} because the event queue is closed.", name);
+ return null;
+ }
+ log.error("Unexpected exception while executing deferred write event {}. " +
+ "Rescheduling for a minute from now.", name, e);
+ scheduleDeferredWriteEvent(name,
+ deadlineNs + TimeUnit.NANOSECONDS.convert(1, TimeUnit.MINUTES), op);
+ return null;
+ });
+ }
+
+ static final String MAYBE_FENCE_REPLICAS = "maybeFenceReplicas";
+
+ class MaybeFenceReplicas implements ControllerWriteOperation<Void> {
+ @Override
+ public ControllerResult<Void> generateRecordsAndResult() {
+ ControllerResult<Set<Integer>> result =
+ clusterControl.maybeFenceLeastRecentlyContacted();
+ for (int brokerId : result.response()) {
+ replicationControl.removeFromIsr(brokerId, result.records());
+ }
+ rescheduleMaybeFenceReplicas();
+ return new ControllerResult<>(result.records(), null);
+ }
+ }
+
+ private void rescheduleMaybeFenceReplicas() {
+ long nextCheckTimeNs = clusterControl.nextCheckTimeNs();
+ if (nextCheckTimeNs == Long.MAX_VALUE) {
+ cancelMaybeFenceReplicas();
+ } else {
+ scheduleDeferredWriteEvent(MAYBE_FENCE_REPLICAS, nextCheckTimeNs,
+ new MaybeFenceReplicas());
+ }
+ }
+
+ private void cancelMaybeFenceReplicas() {
+ queue.cancelDeferred(MAYBE_FENCE_REPLICAS);
+ }
+
+ @SuppressWarnings("unchecked")
+ private void replay(ApiMessage message) {
+ try {
+ MetadataRecordType type = MetadataRecordType.fromId(message.apiKey());
+ switch (type) {
+ case REGISTER_BROKER_RECORD:
+ clusterControl.replay((RegisterBrokerRecord) message);
+ break;
+ case UNREGISTER_BROKER_RECORD:
+ clusterControl.replay((UnregisterBrokerRecord) message);
+ break;
+ case FENCE_BROKER_RECORD:
+ clusterControl.replay((FenceBrokerRecord) message);
+ break;
+ case UNFENCE_BROKER_RECORD:
+ clusterControl.replay((UnfenceBrokerRecord) message);
+ break;
+ case TOPIC_RECORD:
+ replicationControl.replay((TopicRecord) message);
+ break;
+ case PARTITION_RECORD:
+ replicationControl.replay((PartitionRecord) message);
+ break;
+ case CONFIG_RECORD:
+ configurationControl.replay((ConfigRecord) message);
+ break;
+ case QUOTA_RECORD:
+ clientQuotaControlManager.replay((QuotaRecord) message);
+ break;
+ case ISR_CHANGE_RECORD:
+ replicationControl.replay((IsrChangeRecord) message);
+ break;
+ default:
+ throw new RuntimeException("Unhandled record type " + type);
+ }
+ } catch (Exception e) {
+ log.error("Error replaying record {}", message.toString(), e);
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The ID of this controller node.
+ */
+ private final int nodeId;
+
+ /**
+ * The single-threaded queue that processes all of our events.
+ * It also processes timeouts.
+ */
+ private final KafkaEventQueue queue;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * A registry for snapshot data. This must be accessed only by the event queue thread.
+ */
+ private final SnapshotRegistry snapshotRegistry;
+
+ /**
+ * The purgatory which holds deferred operations which are waiting for the metadata
+ * log's high water mark to advance. This must be accessed only by the event queue thread.
+ */
+ private final ControllerPurgatory purgatory;
+
+ /**
+ * An object which stores the controller's dynamic configuration.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * An object which stores the controller's dynamic client quotas.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClientQuotaControlManager clientQuotaControlManager;
+
+ /**
+ * An object which stores the controller's view of the cluster.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ClusterControlManager clusterControl;
+
+ /**
+ * An object which stores the controller's view of the cluster features.
+ * This must be accessed only by the event queue thread.
+ */
+ private final FeatureControlManager featureControl;
+
+ /**
+ * An object which stores the controller's view of topics and partitions.
+ * This must be accessed only by the event queue thread.
+ */
+ private final ReplicationControlManager replicationControl;
+
+ /**
+ * The interface that we use to mutate the Raft log.
+ */
+ private final MetaLogManager logManager;
+
+ /**
+ * The interface that receives callbacks from the Raft log. These callbacks are
+ * invoked from the Raft thread(s), not from the controller thread.
+ */
+ private final QuorumMetaLogListener metaLogListener;
+
+ /**
+ * If this controller is active, this is the non-negative controller epoch.
+ * Otherwise, this is -1. This variable must be modified only from the controller
+ * thread, but it can be read from other threads.
+ */
+ private volatile long curClaimEpoch;
+
+ /**
+ * The last offset we have committed, or -1 if we have not committed any offsets.
+ */
+ private long lastCommittedOffset;
+
+ /**
+ * If we have called scheduleWrite, this is the last offset we got back from it.
+ */
+ private long writeOffset;
+
+ private QuorumController(LogContext logContext,
+ int nodeId,
+ KafkaEventQueue queue,
+ Time time,
+ Map<ConfigResource.Type, ConfigDef> configDefs,
+ MetaLogManager logManager,
+ Map<String, VersionRange> supportedFeatures,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ReplicaPlacementPolicy replicaPlacementPolicy,
+ long sessionTimeoutNs) throws Exception {
+ this.log = logContext.logger(QuorumController.class);
+ this.nodeId = nodeId;
+ this.queue = queue;
+ this.time = time;
+ this.snapshotRegistry = new SnapshotRegistry(logContext);
+ snapshotRegistry.createSnapshot(-1);
+ this.purgatory = new ControllerPurgatory();
+ this.configurationControl = new ConfigurationControlManager(logContext,
+ snapshotRegistry, configDefs);
+ this.clientQuotaControlManager = new ClientQuotaControlManager(snapshotRegistry);
+ this.clusterControl = new ClusterControlManager(logContext, time,
+ snapshotRegistry, sessionTimeoutNs, replicaPlacementPolicy);
+ this.featureControl = new FeatureControlManager(supportedFeatures, snapshotRegistry);
+ this.replicationControl = new ReplicationControlManager(snapshotRegistry,
+ logContext, new Random(), defaultReplicationFactor, defaultNumPartitions,
+ configurationControl, clusterControl);
+ this.logManager = logManager;
+ this.metaLogListener = new QuorumMetaLogListener();
+ this.curClaimEpoch = -1L;
+ this.lastCommittedOffset = -1L;
+ this.writeOffset = -1L;
+ this.logManager.register(metaLogListener);
+ }
+
+ @Override
+ public CompletableFuture<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ return appendWriteEvent("alterIsr", () ->
+ replicationControl.alterIsr(request));
+ }
+
+ @Override
+ public CompletableFuture<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ return appendWriteEvent("createTopics", () ->
+ replicationControl.createTopics(request));
+ }
+
+ @Override
+ public CompletableFuture<Void> decommissionBroker(int brokerId) {
+ return appendWriteEvent("decommissionBroker", () -> {
+ ControllerResult<Void> result = clusterControl.decommissionBroker(brokerId);
+ replicationControl.removeFromIsr(brokerId, result.records());
+ return result;
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ResultOrError<Map<String, String>>>>
+ describeConfigs(Map<ConfigResource, Collection<String>> resources) {
+ return appendReadEvent("describeConfigs", () ->
+ configurationControl.describeConfigs(lastCommittedOffset, resources));
+ }
+
+ @Override
+ public CompletableFuture<ElectLeadersResponseData>
+ electLeaders(ElectLeadersRequestData request) {
+ CompletableFuture<ElectLeadersResponseData> future = new CompletableFuture<>();
+ appendWriteEvent("electLeaders", request.timeoutMs(),
+ () -> replicationControl.electLeaders(request));
+ return future;
+ }
+
+ @Override
+ public CompletableFuture<FeatureManager.FinalizedFeaturesAndEpoch> finalizedFeatures() {
+ return appendReadEvent("getFinalizedFeatures",
+ () -> featureControl.finalizedFeaturesAndEpoch(lastCommittedOffset));
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> incrementalAlterConfigs(
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges,
+ boolean validateOnly) {
+ return appendWriteEvent("incrementalAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ @Override
+ public CompletableFuture<Map<ConfigResource, ApiError>> legacyAlterConfigs(
+ Map<ConfigResource, Map<String, String>> newConfigs, boolean validateOnly) {
+ return appendWriteEvent("legacyAlterConfigs", () -> {
+ ControllerResult<Map<ConfigResource, ApiError>> result =
+ configurationControl.legacyAlterConfigs(newConfigs);
+ if (validateOnly) {
+ return result.withoutRecords();
+ } else {
+ return result;
+ }
+ });
+ }
+
+ class ProcessBrokerHeartbeatOperation
+ implements ControllerWriteOperation<BrokerHeartbeatReply> {
+ private final BrokerHeartbeatRequestData request;
+ private boolean updateShutdownOffset = false;
+
+ ProcessBrokerHeartbeatOperation(BrokerHeartbeatRequestData request) {
+ this.request = request;
+ }
+
+ @Override
+ public ControllerResult<BrokerHeartbeatReply> generateRecordsAndResult() {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ boolean movingLeadersForShutDown = false;
+ if (request.wantShutDown()) {
+ records = replicationControl.removeLeaderships(request.brokerId());
+ movingLeadersForShutDown = !records.isEmpty();
+ }
+ ControllerResult<BrokerHeartbeatReply> result = clusterControl.
+ processBrokerHeartbeat(request, lastCommittedOffset,
+ movingLeadersForShutDown);
+ if (movingLeadersForShutDown && !result.response().shouldShutDown()) {
+ updateShutdownOffset = true;
+ }
+ if (result.response().isFenced()) {
+ replicationControl.removeFromIsr(request.brokerId(), records);
Review comment:
It seems that we if the request wants to shut down, we should always remove the shutting down broker from isr, just like moving the leader off the shutting down broker?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
rondagostino commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r573093783
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
Review comment:
Should the second check appear within the first `if` as it does below in `touch()`? And assuming yes, maybe refactor that common logic out into a `private void removeFromActiveAndUnfenced(BrokerHeartbeatState broker)` method?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
Review comment:
What is supposed to happen if it is already shutting down and this is invoked? Will it matter if `deferred` is different in the second call?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
Review comment:
`boolean hasValidSession(...)`?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
Review comment:
This seems to imply that it is impossible for a broker to be doing a controlled shutdown and be fenced. I guess that means any controlled shutdown gets cancelled? A comment explaining the implications would be helpful. Actually, from further down in `shouldShutDown()` it appears it can shutdown immediately if it is fenced -- so I think it's about leaders moving away? Again, a comment would help.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
Review comment:
At first I was confused as to why these two operations were necessary, then I realized it is because the instance is mutable and its places in the ordered list and `TreeSet` are going to change. A comment here would be helpful to make this apparent (I know there is a comment in the list and set declarations, but a reminder here would be helpful nonetheless).
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
Review comment:
`public void remove(...)`?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
Review comment:
`public void touch(...)`?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
Review comment:
`public void beginBrokerShutDown(...)`? Javadoc would be helpful, especially to explain what `deferred` is about. Or, if `private` rather than `public`, at least a comment.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
Review comment:
`public` or `private`? Same with methods below.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
Review comment:
You seem to sometime use `shutdown` and other times use `shutDown` -- not sure if that is on purpose or there is a lack of consistency?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
+ if (!broker.shuttingDown()) {
+ log.trace("broker {} should not shut down because it is not in the " +
+ "shutdown state.", brokerId);
+ return false;
+ }
+ long curOffset = lowestActiveOffset();
Review comment:
`currLowestActiveOffset` a better name?
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
Review comment:
Is this the case because fenced implies leadership is already moving away? If so, a comment to that effect (or some additional wording in the log line) would be helpful.
##########
File path: metadata/src/main/java/org/apache/kafka/controller/BrokerHeartbeatManager.java
##########
@@ -0,0 +1,491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.metadata.UsableBroker;
+import org.slf4j.Logger;
+
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.TreeSet;
+import java.util.function.Function;
+
+
+/**
+ * The BrokerHeartbeatManager manages all the soft state associated with broker heartbeats.
+ * Soft state is state which does not appear in the metadata log. This state includes
+ * things like the last time each broker sent us a heartbeat, and whether the broker is
+ * trying to perform a controlled shutdown.
+ *
+ * Only the active controller has a BrokerHeartbeatManager, since only the active
+ * controller handles broker heartbeats. Standby controllers will create a heartbeat
+ * manager as part of the process of activating. This design minimizes the size of the
+ * metadata partition by excluding heartbeats from it. However, it does mean that after
+ * a controller failover, we may take some extra time to fence brokers, since the new
+ * active controller does not know when the last heartbeats were received from each.
+ */
+public class BrokerHeartbeatManager {
+ static class BrokerHeartbeatState {
+ /**
+ * The broker ID.
+ */
+ private final int id;
+
+ /**
+ * The last time we received a heartbeat from this broker, in monotonic nanoseconds.
+ * When this field is updated, we also may have to update the broker's position in
+ * the unfenced list.
+ */
+ long lastContactNs;
+
+ /**
+ * The last metadata offset which this broker reported. When this field is updated,
+ * we may also have to update the broker's position in the active set.
+ */
+ long metadataOffset;
+
+ /**
+ * The offset at which the broker should complete its controlled shutdown, or -1
+ * if the broker is not performing a controlled shutdown. When this field is
+ * updated, we also have to update the broker's position in the shuttingDown set.
+ */
+ private long shutdownOffset;
+
+ /**
+ * The previous entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState prev;
+
+ /**
+ * The next entry in the unfenced list, or null if the broker is not in that list.
+ */
+ private BrokerHeartbeatState next;
+
+ BrokerHeartbeatState(int id) {
+ this.id = id;
+ this.lastContactNs = 0;
+ this.prev = null;
+ this.next = null;
+ this.metadataOffset = -1;
+ this.shutdownOffset = -1;
+ }
+
+ /**
+ * Returns the broker ID.
+ */
+ int id() {
+ return id;
+ }
+
+ /**
+ * Returns true only if the broker is fenced.
+ */
+ boolean fenced() {
+ return prev == null;
+ }
+
+ /**
+ * Returns true only if the broker is in controlled shutdown state.
+ */
+ boolean shuttingDown() {
+ return shutdownOffset >= 0;
+ }
+ }
+
+ static class MetadataOffsetComparator implements Comparator<BrokerHeartbeatState> {
+ static final MetadataOffsetComparator INSTANCE = new MetadataOffsetComparator();
+
+ @Override
+ public int compare(BrokerHeartbeatState a, BrokerHeartbeatState b) {
+ if (a.metadataOffset < b.metadataOffset) {
+ return -1;
+ } else if (a.metadataOffset > b.metadataOffset) {
+ return 1;
+ } else if (a.id < b.id) {
+ return -1;
+ } else if (a.id > b.id) {
+ return 1;
+ } else {
+ return 0;
+ }
+ }
+ }
+
+ static class BrokerHeartbeatStateList {
+ /**
+ * The head of the list of unfenced brokers. The list is sorted in ascending order
+ * of last contact time.
+ */
+ private final BrokerHeartbeatState head;
+
+ BrokerHeartbeatStateList() {
+ this.head = new BrokerHeartbeatState(-1);
+ head.prev = head;
+ head.next = head;
+ }
+
+ /**
+ * Return the head of the list, or null if the list is empty.
+ */
+ BrokerHeartbeatState first() {
+ BrokerHeartbeatState result = head.next;
+ return result == head ? null : result;
+ }
+
+ /**
+ * Add the broker to the list. We start looking for a place to put it at the end
+ * of the list.
+ */
+ void add(BrokerHeartbeatState broker) {
+ BrokerHeartbeatState cur = head.prev;
+ while (true) {
+ if (cur == head || cur.lastContactNs <= broker.lastContactNs) {
+ broker.next = cur.next;
+ cur.next.prev = broker;
+ broker.prev = cur;
+ cur.next = broker;
+ break;
+ }
+ cur = cur.prev;
+ }
+ }
+
+ /**
+ * Remove a broker from the list.
+ */
+ void remove(BrokerHeartbeatState broker) {
+ if (broker.next == null) {
+ throw new RuntimeException(broker + " is not in the list.");
+ }
+ broker.prev.next = broker.next;
+ broker.next.prev = broker.prev;
+ broker.prev = null;
+ broker.next = null;
+ }
+
+ BrokerHeartbeatStateIterator iterator() {
+ return new BrokerHeartbeatStateIterator(head);
+ }
+ }
+
+ static class BrokerHeartbeatStateIterator implements Iterator<BrokerHeartbeatState> {
+ private final BrokerHeartbeatState head;
+ private BrokerHeartbeatState cur;
+
+ BrokerHeartbeatStateIterator(BrokerHeartbeatState head) {
+ this.head = head;
+ this.cur = head;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return cur.next != head;
+ }
+
+ @Override
+ public BrokerHeartbeatState next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ BrokerHeartbeatState result = cur.next;
+ cur = cur.next;
+ return result;
+ }
+ }
+
+ private final Logger log;
+
+ /**
+ * The Kafka clock object to use.
+ */
+ private final Time time;
+
+ /**
+ * The broker session timeout in nanoseconds.
+ */
+ private final long sessionTimeoutNs;
+
+ /**
+ * Maps broker IDs to heartbeat states.
+ */
+ private final HashMap<Integer, BrokerHeartbeatState> brokers;
+
+ /**
+ * The list of unfenced brokers, sorted by last contact time.
+ */
+ private final BrokerHeartbeatStateList unfenced;
+
+ /**
+ * The set of active brokers. A broker is active if it is unfenced, and not shutting
+ * down.
+ */
+ private final TreeSet<BrokerHeartbeatState> active;
+
+ BrokerHeartbeatManager(LogContext logContext,
+ Time time,
+ long sessionTimeoutNs) {
+ this.log = logContext.logger(ClusterControlManager.class);
+ this.time = time;
+ this.sessionTimeoutNs = sessionTimeoutNs;
+ this.brokers = new HashMap<>();
+ this.unfenced = new BrokerHeartbeatStateList();
+ this.active = new TreeSet<>(MetadataOffsetComparator.INSTANCE);
+ }
+
+ // VisibleForTesting
+ Time time() {
+ return time;
+ }
+
+ // VisibleForTesting
+ BrokerHeartbeatStateList unfenced() {
+ return unfenced;
+ }
+
+ /**
+ * Remove broker state.
+ *
+ * @param brokerId The broker ID to remove.
+ */
+ void remove(int brokerId) {
+ BrokerHeartbeatState broker = brokers.remove(brokerId);
+ if (broker != null) {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ }
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param brokerId The broker ID to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ boolean hasValidSession(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) return false;
+ return hasValidSession(broker);
+ }
+
+ /**
+ * Check if the given broker has a valid session.
+ *
+ * @param broker The broker to check.
+ *
+ * @return True if the given broker has a valid session.
+ */
+ private boolean hasValidSession(BrokerHeartbeatState broker) {
+ if (broker.fenced()) {
+ return false;
+ } else {
+ return broker.lastContactNs + sessionTimeoutNs >= time.nanoseconds();
+ }
+ }
+
+ /**
+ * Update broker state, including lastContactNs.
+ *
+ * @param brokerId The broker ID.
+ * @param fenced True only if the broker is currently fenced.
+ * @param metadataOffset The latest metadata offset of the broker.
+ */
+ void touch(int brokerId, boolean fenced, long metadataOffset) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ broker = new BrokerHeartbeatState(brokerId);
+ brokers.put(brokerId, broker);
+ } else {
+ if (!broker.fenced()) {
+ unfenced.remove(broker);
+ if (!broker.shuttingDown()) {
+ active.remove(broker);
+ }
+ }
+ }
+ broker.lastContactNs = time.nanoseconds();
+ broker.metadataOffset = metadataOffset;
+ if (fenced) {
+ broker.shutdownOffset = -1;
+ } else {
+ unfenced.add(broker);
+ if (!broker.shuttingDown()) {
+ active.add(broker);
+ }
+ }
+ }
+
+ void beginBrokerShutDown(int brokerId, boolean deferred) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (!broker.shuttingDown()) {
+ log.debug("broker {} has entered controlled shutdown.", brokerId);
+ active.remove(broker);
+ broker.shutdownOffset = deferred ? Long.MAX_VALUE : 0;
+ }
+ }
+
+ boolean shouldShutDown(int brokerId) {
+ BrokerHeartbeatState broker = brokers.get(brokerId);
+ if (broker == null) {
+ throw new RuntimeException("Unable to locate broker " + brokerId);
+ }
+ if (broker.fenced()) {
+ log.debug("broker {} can shut down immediately because it is fenced.");
+ return true;
+ }
+ if (!broker.shuttingDown()) {
+ log.trace("broker {} should not shut down because it is not in the " +
+ "shutdown state.", brokerId);
+ return false;
+ }
+ long curOffset = lowestActiveOffset();
+ if (curOffset >= broker.shutdownOffset) {
+ log.debug("broker {} should shut down. The current lowest active offset, {} " +
+ "is greater or equal to the broker shutdown offset of {}", brokerId,
+ curOffset, broker.shutdownOffset);
+ return true;
+ } else {
+ log.debug("broker {} must wait to shut down. The current lowest active offset, {} " +
+ "is lower than the broker shutdown offset of {}", brokerId,
+ curOffset, broker.shutdownOffset);
+ return false;
+ }
+ }
+
+ long lowestActiveOffset() {
+ Iterator<BrokerHeartbeatState> iterator = active.iterator();
+ if (!iterator.hasNext()) {
+ return Long.MAX_VALUE;
+ }
+ BrokerHeartbeatState first = iterator.next();
+ return first.metadataOffset;
+ }
+
+ void updateShutdownOffset(int brokerId, long offset) {
Review comment:
What's the difference between `beginBrokerShutDown()` and `updateShutdownOffset()`? Why would the offset at which it can shutdown change? A comment would be helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a change in pull request #10070: KAFKA-12276: Add the quorum controller code
Posted by GitBox <gi...@apache.org>.
cmccabe commented on a change in pull request #10070:
URL: https://github.com/apache/kafka/pull/10070#discussion_r579547098
##########
File path: metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.controller;
+
+import org.apache.kafka.clients.admin.AlterConfigOp.OpType;
+import org.apache.kafka.common.ElectionType;
+import org.apache.kafka.common.Uuid;
+import org.apache.kafka.common.config.ConfigResource;
+import org.apache.kafka.common.errors.BrokerIdNotRegisteredException;
+import org.apache.kafka.common.errors.InvalidReplicationFactorException;
+import org.apache.kafka.common.errors.InvalidRequestException;
+import org.apache.kafka.common.errors.InvalidTopicException;
+import org.apache.kafka.common.internals.Topic;
+import org.apache.kafka.common.message.AlterIsrRequestData;
+import org.apache.kafka.common.message.AlterIsrResponseData;
+import org.apache.kafka.common.message.BrokerHeartbeatRequestData;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableReplicaAssignment;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopic;
+import org.apache.kafka.common.message.CreateTopicsRequestData.CreatableTopicCollection;
+import org.apache.kafka.common.message.CreateTopicsRequestData;
+import org.apache.kafka.common.message.CreateTopicsResponseData.CreatableTopicResult;
+import org.apache.kafka.common.message.CreateTopicsResponseData;
+import org.apache.kafka.common.message.ElectLeadersRequestData.TopicPartitions;
+import org.apache.kafka.common.message.ElectLeadersRequestData;
+import org.apache.kafka.common.message.ElectLeadersResponseData.PartitionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData.ReplicaElectionResult;
+import org.apache.kafka.common.message.ElectLeadersResponseData;
+import org.apache.kafka.common.metadata.FenceBrokerRecord;
+import org.apache.kafka.common.metadata.PartitionChangeRecord;
+import org.apache.kafka.common.metadata.PartitionRecord;
+import org.apache.kafka.common.metadata.TopicRecord;
+import org.apache.kafka.common.metadata.UnfenceBrokerRecord;
+import org.apache.kafka.common.metadata.UnregisterBrokerRecord;
+import org.apache.kafka.common.protocol.Errors;
+import org.apache.kafka.common.requests.ApiError;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.controller.BrokersToIsrs.TopicPartition;
+import org.apache.kafka.metadata.ApiMessageAndVersion;
+import org.apache.kafka.metadata.BrokerHeartbeatReply;
+import org.apache.kafka.metadata.BrokerRegistration;
+import org.apache.kafka.timeline.SnapshotRegistry;
+import org.apache.kafka.timeline.TimelineHashMap;
+import org.slf4j.Logger;
+
+import java.util.AbstractMap.SimpleImmutableEntry;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Random;
+
+import static org.apache.kafka.clients.admin.AlterConfigOp.OpType.SET;
+import static org.apache.kafka.common.config.ConfigResource.Type.TOPIC;
+
+
+/**
+ * The ReplicationControlManager is the part of the controller which deals with topics
+ * and partitions. It is responsible for managing the in-sync replica set and leader
+ * of each partition, as well as administrative tasks like creating or deleting topics.
+ */
+public class ReplicationControlManager {
+ /**
+ * A special value used to represent the leader for a partition with no leader.
+ */
+ public static final int NO_LEADER = -1;
+
+ /**
+ * A special value used to represent a PartitionChangeRecord that does not change the
+ * partition leader.
+ */
+ public static final int NO_LEADER_CHANGE = -2;
+
+ static class TopicControlInfo {
+ private final Uuid id;
+ private final TimelineHashMap<Integer, PartitionControlInfo> parts;
+
+ TopicControlInfo(SnapshotRegistry snapshotRegistry, Uuid id) {
+ this.id = id;
+ this.parts = new TimelineHashMap<>(snapshotRegistry, 0);
+ }
+ }
+
+ static class PartitionControlInfo {
+ private final int[] replicas;
+ private final int[] isr;
+ private final int[] removingReplicas;
+ private final int[] addingReplicas;
+ private final int leader;
+ private final int leaderEpoch;
+ private final int partitionEpoch;
+
+ PartitionControlInfo(PartitionRecord record) {
+ this(Replicas.toArray(record.replicas()),
+ Replicas.toArray(record.isr()),
+ Replicas.toArray(record.removingReplicas()),
+ Replicas.toArray(record.addingReplicas()),
+ record.leader(),
+ record.leaderEpoch(),
+ record.partitionEpoch());
+ }
+
+ PartitionControlInfo(int[] replicas, int[] isr, int[] removingReplicas,
+ int[] addingReplicas, int leader, int leaderEpoch,
+ int partitionEpoch) {
+ this.replicas = replicas;
+ this.isr = isr;
+ this.removingReplicas = removingReplicas;
+ this.addingReplicas = addingReplicas;
+ this.leader = leader;
+ this.leaderEpoch = leaderEpoch;
+ this.partitionEpoch = partitionEpoch;
+ }
+
+ PartitionControlInfo merge(PartitionChangeRecord record) {
+ int[] newIsr = (record.isr() == null) ? isr : Replicas.toArray(record.isr());
+ int newLeader;
+ int newLeaderEpoch;
+ if (record.leader() == NO_LEADER_CHANGE) {
+ newLeader = leader;
+ newLeaderEpoch = leaderEpoch;
+ } else {
+ newLeader = record.leader();
+ newLeaderEpoch = leaderEpoch + 1;
+ }
+ return new PartitionControlInfo(replicas,
+ newIsr,
+ removingReplicas,
+ addingReplicas,
+ newLeader,
+ newLeaderEpoch,
+ partitionEpoch + 1);
+ }
+
+ String diff(PartitionControlInfo prev) {
+ StringBuilder builder = new StringBuilder();
+ String prefix = "";
+ if (!Arrays.equals(replicas, prev.replicas)) {
+ builder.append(prefix).append("oldReplicas=").append(Arrays.toString(prev.replicas));
+ prefix = ", ";
+ builder.append(prefix).append("newReplicas=").append(Arrays.toString(replicas));
+ }
+ if (!Arrays.equals(isr, prev.isr)) {
+ builder.append(prefix).append("oldIsr=").append(Arrays.toString(prev.isr));
+ prefix = ", ";
+ builder.append(prefix).append("newIsr=").append(Arrays.toString(isr));
+ }
+ if (!Arrays.equals(removingReplicas, prev.removingReplicas)) {
+ builder.append(prefix).append("oldRemovingReplicas=").
+ append(Arrays.toString(prev.removingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newRemovingReplicas=").
+ append(Arrays.toString(removingReplicas));
+ }
+ if (!Arrays.equals(addingReplicas, prev.addingReplicas)) {
+ builder.append(prefix).append("oldAddingReplicas=").
+ append(Arrays.toString(prev.addingReplicas));
+ prefix = ", ";
+ builder.append(prefix).append("newAddingReplicas=").
+ append(Arrays.toString(addingReplicas));
+ }
+ if (leader != prev.leader) {
+ builder.append(prefix).append("oldLeader=").append(prev.leader);
+ prefix = ", ";
+ builder.append(prefix).append("newLeader=").append(leader);
+ }
+ if (leaderEpoch != prev.leaderEpoch) {
+ builder.append(prefix).append("oldLeaderEpoch=").append(prev.leaderEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newLeaderEpoch=").append(leaderEpoch);
+ }
+ if (partitionEpoch != prev.partitionEpoch) {
+ builder.append(prefix).append("oldPartitionEpoch=").append(prev.partitionEpoch);
+ prefix = ", ";
+ builder.append(prefix).append("newPartitionEpoch=").append(partitionEpoch);
+ }
+ return builder.toString();
+ }
+
+ boolean hasLeader() {
+ return leader != NO_LEADER;
+ }
+
+ int preferredReplica() {
+ return replicas.length == 0 ? NO_LEADER : replicas[0];
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(replicas, isr, removingReplicas, addingReplicas, leader,
+ leaderEpoch, partitionEpoch);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (!(o instanceof PartitionControlInfo)) return false;
+ PartitionControlInfo other = (PartitionControlInfo) o;
+ return diff(other).isEmpty();
+ }
+
+ @Override
+ public String toString() {
+ StringBuilder builder = new StringBuilder("PartitionControlInfo(");
+ builder.append("replicas=").append(Arrays.toString(replicas));
+ builder.append(", isr=").append(Arrays.toString(isr));
+ builder.append(", removingReplicas=").append(Arrays.toString(removingReplicas));
+ builder.append(", addingReplicas=").append(Arrays.toString(addingReplicas));
+ builder.append(", leader=").append(leader);
+ builder.append(", leaderEpoch=").append(leaderEpoch);
+ builder.append(", partitionEpoch=").append(partitionEpoch);
+ builder.append(")");
+ return builder.toString();
+ }
+ }
+
+ private final SnapshotRegistry snapshotRegistry;
+ private final Logger log;
+
+ /**
+ * The random number generator used by this object.
+ */
+ private final Random random;
+
+ /**
+ * The KIP-464 default replication factor that is used if a CreateTopics request does
+ * not specify one.
+ */
+ private final short defaultReplicationFactor;
+
+ /**
+ * The KIP-464 default number of partitions that is used if a CreateTopics request does
+ * not specify a number of partitions.
+ */
+ private final int defaultNumPartitions;
+
+ /**
+ * A reference to the controller's configuration control manager.
+ */
+ private final ConfigurationControlManager configurationControl;
+
+ /**
+ * A reference to the controller's cluster control manager.
+ */
+ final ClusterControlManager clusterControl;
+
+ /**
+ * Maps topic names to topic UUIDs.
+ */
+ private final TimelineHashMap<String, Uuid> topicsByName;
+
+ /**
+ * Maps topic UUIDs to structures containing topic information, including partitions.
+ */
+ private final TimelineHashMap<Uuid, TopicControlInfo> topics;
+
+ /**
+ * A map of broker IDs to the partitions that the broker is in the ISR for.
+ */
+ private final BrokersToIsrs brokersToIsrs;
+
+ ReplicationControlManager(SnapshotRegistry snapshotRegistry,
+ LogContext logContext,
+ Random random,
+ short defaultReplicationFactor,
+ int defaultNumPartitions,
+ ConfigurationControlManager configurationControl,
+ ClusterControlManager clusterControl) {
+ this.snapshotRegistry = snapshotRegistry;
+ this.log = logContext.logger(ReplicationControlManager.class);
+ this.random = random;
+ this.defaultReplicationFactor = defaultReplicationFactor;
+ this.defaultNumPartitions = defaultNumPartitions;
+ this.configurationControl = configurationControl;
+ this.clusterControl = clusterControl;
+ this.topicsByName = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.topics = new TimelineHashMap<>(snapshotRegistry, 0);
+ this.brokersToIsrs = new BrokersToIsrs(snapshotRegistry);
+ }
+
+ public void replay(TopicRecord record) {
+ topicsByName.put(record.name(), record.topicId());
+ topics.put(record.topicId(), new TopicControlInfo(snapshotRegistry, record.topicId()));
+ log.info("Created topic {} with ID {}.", record.name(), record.topicId());
+ }
+
+ public void replay(PartitionRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo newPartInfo = new PartitionControlInfo(record);
+ PartitionControlInfo prevPartInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartInfo == null) {
+ log.info("Created partition {}:{} with {}.", record.topicId(),
+ record.partitionId(), newPartInfo.toString());
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(), null,
+ newPartInfo.isr, NO_LEADER, newPartInfo.leader);
+ } else {
+ String diff = newPartInfo.diff(prevPartInfo);
+ if (!diff.isEmpty()) {
+ log.info("Modified partition {}:{}: {}.", record.topicId(),
+ record.partitionId(), diff);
+ topicInfo.parts.put(record.partitionId(), newPartInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartInfo.isr, newPartInfo.isr, prevPartInfo.leader,
+ newPartInfo.leader);
+ }
+ }
+ }
+
+ public void replay(PartitionChangeRecord record) {
+ TopicControlInfo topicInfo = topics.get(record.topicId());
+ if (topicInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no topic with that ID was found.");
+ }
+ PartitionControlInfo prevPartitionInfo = topicInfo.parts.get(record.partitionId());
+ if (prevPartitionInfo == null) {
+ throw new RuntimeException("Tried to create partition " + record.topicId() +
+ ":" + record.partitionId() + ", but no partition with that id was found.");
+ }
+ PartitionControlInfo newPartitionInfo = prevPartitionInfo.merge(record);
+ topicInfo.parts.put(record.partitionId(), newPartitionInfo);
+ brokersToIsrs.update(record.topicId(), record.partitionId(),
+ prevPartitionInfo.isr, newPartitionInfo.isr, prevPartitionInfo.leader,
+ newPartitionInfo.leader);
+ log.debug("Applied ISR change record: {}", record.toString());
+ }
+
+ ControllerResult<CreateTopicsResponseData>
+ createTopics(CreateTopicsRequestData request) {
+ Map<String, ApiError> topicErrors = new HashMap<>();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+
+ // Check the topic names.
+ validateNewTopicNames(topicErrors, request.topics());
+
+ // Identify topics that already exist and mark them with the appropriate error
+ request.topics().stream().filter(creatableTopic -> topicsByName.containsKey(creatableTopic.name()))
+ .forEach(t -> topicErrors.put(t.name(), new ApiError(Errors.TOPIC_ALREADY_EXISTS)));
+
+ // Verify that the configurations for the new topics are OK, and figure out what
+ // ConfigRecords should be created.
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges =
+ computeConfigChanges(topicErrors, request.topics());
+ ControllerResult<Map<ConfigResource, ApiError>> configResult =
+ configurationControl.incrementalAlterConfigs(configChanges);
+ for (Entry<ConfigResource, ApiError> entry : configResult.response().entrySet()) {
+ if (entry.getValue().isFailure()) {
+ topicErrors.put(entry.getKey().name(), entry.getValue());
+ }
+ }
+ records.addAll(configResult.records());
+
+ // Try to create whatever topics are needed.
+ Map<String, CreatableTopicResult> successes = new HashMap<>();
+ for (CreatableTopic topic : request.topics()) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ ApiError error = createTopic(topic, records, successes);
+ if (error.isFailure()) {
+ topicErrors.put(topic.name(), error);
+ }
+ }
+
+ // Create responses for all topics.
+ CreateTopicsResponseData data = new CreateTopicsResponseData();
+ StringBuilder resultsBuilder = new StringBuilder();
+ String resultsPrefix = "";
+ for (CreatableTopic topic : request.topics()) {
+ ApiError error = topicErrors.get(topic.name());
+ if (error != null) {
+ data.topics().add(new CreatableTopicResult().
+ setName(topic.name()).
+ setErrorCode(error.error().code()).
+ setErrorMessage(error.message()));
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append(error.error()).append(" (").append(error.message()).append(")");
+ resultsPrefix = ", ";
+ continue;
+ }
+ CreatableTopicResult result = successes.get(topic.name());
+ data.topics().add(result);
+ resultsBuilder.append(resultsPrefix).append(topic).append(": ").
+ append("SUCCESS");
+ resultsPrefix = ", ";
+ }
+ log.info("createTopics result(s): {}", resultsBuilder.toString());
+ return new ControllerResult<>(records, data);
+ }
+
+ private ApiError createTopic(CreatableTopic topic,
+ List<ApiMessageAndVersion> records,
+ Map<String, CreatableTopicResult> successes) {
+ Map<Integer, PartitionControlInfo> newParts = new HashMap<>();
+ if (!topic.assignments().isEmpty()) {
+ if (topic.replicationFactor() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but replication " +
+ "factor was not set to -1.");
+ }
+ if (topic.numPartitions() != -1) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "A manual partition assignment was specified, but numPartitions " +
+ "was not set to -1.");
+ }
+ for (CreatableReplicaAssignment assignment : topic.assignments()) {
+ if (newParts.containsKey(assignment.partitionIndex())) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "Found multiple manual partition assignments for partition " +
+ assignment.partitionIndex());
+ }
+ HashSet<Integer> brokerIds = new HashSet<>();
+ for (int brokerId : assignment.brokerIds()) {
+ if (!brokerIds.add(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment specifies the same node " +
+ "id more than once.");
+ } else if (!clusterControl.unfenced(brokerId)) {
+ return new ApiError(Errors.INVALID_REPLICA_ASSIGNMENT,
+ "The manual partition assignment contains node " + brokerId +
+ ", but that node is not usable.");
+ }
+ }
+ int[] replicas = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ replicas[i] = assignment.brokerIds().get(i);
+ }
+ int[] isr = new int[assignment.brokerIds().size()];
+ for (int i = 0; i < replicas.length; i++) {
+ isr[i] = assignment.brokerIds().get(i);
+ }
+ newParts.put(assignment.partitionIndex(),
+ new PartitionControlInfo(replicas, isr, null, null, isr[0], 0, 0));
+ }
+ } else if (topic.replicationFactor() < -1 || topic.replicationFactor() == 0) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Replication factor was set to an invalid non-positive value.");
+ } else if (!topic.assignments().isEmpty()) {
+ return new ApiError(Errors.INVALID_REQUEST,
+ "Replication factor was not set to -1 but a manual partition " +
+ "assignment was specified.");
+ } else if (topic.numPartitions() < -1 || topic.numPartitions() == 0) {
+ return new ApiError(Errors.INVALID_PARTITIONS,
+ "Number of partitions was set to an invalid non-positive value.");
+ } else {
+ int numPartitions = topic.numPartitions() == -1 ?
+ defaultNumPartitions : topic.numPartitions();
+ short replicationFactor = topic.replicationFactor() == -1 ?
+ defaultReplicationFactor : topic.replicationFactor();
+ try {
+ List<List<Integer>> replicas = clusterControl.
+ placeReplicas(numPartitions, replicationFactor);
+ for (int partitionId = 0; partitionId < replicas.size(); partitionId++) {
+ int[] r = Replicas.toArray(replicas.get(partitionId));
+ newParts.put(partitionId,
+ new PartitionControlInfo(r, r, null, null, r[0], 0, 0));
+ }
+ } catch (InvalidReplicationFactorException e) {
+ return new ApiError(Errors.INVALID_REPLICATION_FACTOR,
+ "Unable to replicate the partition " + replicationFactor +
+ " times: " + e.getMessage());
+ }
+ }
+ Uuid topicId = new Uuid(random.nextLong(), random.nextLong());
+ successes.put(topic.name(), new CreatableTopicResult().
+ setName(topic.name()).
+ setTopicId(topicId).
+ setErrorCode((short) 0).
+ setErrorMessage(null).
+ setNumPartitions(newParts.size()).
+ setReplicationFactor((short) newParts.get(0).replicas.length));
+ records.add(new ApiMessageAndVersion(new TopicRecord().
+ setName(topic.name()).
+ setTopicId(topicId), (short) 0));
+ for (Entry<Integer, PartitionControlInfo> partEntry : newParts.entrySet()) {
+ int partitionIndex = partEntry.getKey();
+ PartitionControlInfo info = partEntry.getValue();
+ records.add(new ApiMessageAndVersion(new PartitionRecord().
+ setPartitionId(partitionIndex).
+ setTopicId(topicId).
+ setReplicas(Replicas.toList(info.replicas)).
+ setIsr(Replicas.toList(info.isr)).
+ setRemovingReplicas(null).
+ setAddingReplicas(null).
+ setLeader(info.leader).
+ setLeaderEpoch(info.leaderEpoch).
+ setPartitionEpoch(0), (short) 0));
+ }
+ return ApiError.NONE;
+ }
+
+ static void validateNewTopicNames(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ try {
+ Topic.validate(topic.name());
+ } catch (InvalidTopicException e) {
+ topicErrors.put(topic.name(),
+ new ApiError(Errors.INVALID_TOPIC_EXCEPTION, e.getMessage()));
+ }
+ }
+ }
+
+ static Map<ConfigResource, Map<String, Entry<OpType, String>>>
+ computeConfigChanges(Map<String, ApiError> topicErrors,
+ CreatableTopicCollection topics) {
+ Map<ConfigResource, Map<String, Entry<OpType, String>>> configChanges = new HashMap<>();
+ for (CreatableTopic topic : topics) {
+ if (topicErrors.containsKey(topic.name())) continue;
+ Map<String, Entry<OpType, String>> topicConfigs = new HashMap<>();
+ for (CreateTopicsRequestData.CreateableTopicConfig config : topic.configs()) {
+ topicConfigs.put(config.name(), new SimpleImmutableEntry<>(SET, config.value()));
+ }
+ if (!topicConfigs.isEmpty()) {
+ configChanges.put(new ConfigResource(TOPIC, topic.name()), topicConfigs);
+ }
+ }
+ return configChanges;
+ }
+
+ // VisibleForTesting
+ PartitionControlInfo getPartition(Uuid topicId, int partitionId) {
+ TopicControlInfo topic = topics.get(topicId);
+ if (topic == null) {
+ return null;
+ }
+ return topic.parts.get(partitionId);
+ }
+
+ // VisibleForTesting
+ BrokersToIsrs brokersToIsrs() {
+ return brokersToIsrs;
+ }
+
+ ControllerResult<AlterIsrResponseData> alterIsr(AlterIsrRequestData request) {
+ clusterControl.checkBrokerEpoch(request.brokerId(), request.brokerEpoch());
+ AlterIsrResponseData response = new AlterIsrResponseData();
+ List<ApiMessageAndVersion> records = new ArrayList<>();
+ for (AlterIsrRequestData.TopicData topicData : request.topics()) {
+ AlterIsrResponseData.TopicData responseTopicData =
+ new AlterIsrResponseData.TopicData().setName(topicData.name());
+ response.topics().add(responseTopicData);
+ Uuid topicId = topicsByName.get(topicData.name());
+ if (topicId == null || !topics.containsKey(topicId)) {
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ }
+ continue;
+ }
+ TopicControlInfo topic = topics.get(topicId);
+ for (AlterIsrRequestData.PartitionData partitionData : topicData.partitions()) {
+ PartitionControlInfo partition = topic.parts.get(partitionData.partitionIndex());
+ if (partition == null) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.UNKNOWN_TOPIC_OR_PARTITION.code()));
+ continue;
+ }
+ if (partitionData.leaderEpoch() != partition.leaderEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.FENCED_LEADER_EPOCH.code()));
+ continue;
+ }
+ if (partitionData.currentIsrVersion() != partition.partitionEpoch) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_UPDATE_VERSION.code()));
+ continue;
+ }
+ int[] newIsr = Replicas.toArray(partitionData.newIsr());
+ if (!Replicas.validateIsr(partition.replicas, newIsr)) {
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ continue;
+ }
+ if (!Replicas.contains(newIsr, partition.leader)) {
+ // An alterIsr request can't remove the current leader.
+ responseTopicData.partitions().add(new AlterIsrResponseData.PartitionData().
+ setPartitionIndex(partitionData.partitionIndex()).
+ setErrorCode(Errors.INVALID_REQUEST.code()));
+ continue;
+ }
+ records.add(new ApiMessageAndVersion(new PartitionChangeRecord().
Review comment:
That is the default so we don't need to set it unless we're changing it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org