You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "DanielWang2035 (via GitHub)" <gi...@apache.org> on 2026/03/02 02:29:25 UTC
[PR] Subscription: implement IoTConsensus-based subscription [iotdb]
DanielWang2035 opened a new pull request, #17238:
URL: https://github.com/apache/iotdb/pull/17238
## [Draft] Subscription: Implement IoTConsensus-based subscription
### Description
Adds a consensus-based subscription path that reads directly from IoTConsensus WAL, bypassing the Pipe framework. Activated for live-mode, Tablet-format topics (`mode=live AND format≠TsFileHandler`). No changes to existing Pipe-based subscription behavior.
### How It Works
1. `IoTConsensusServerImpl.write()` offers `IndexedConsensusRequest` to registered in-memory queues (non-blocking, drop-on-full)
2. Background `prefetchLoop` thread drains pending entries, fills WAL gaps, deserializes InsertNode, converts to Tablet via `ConsensusLogToTabletConverter`, and enqueues `SubscriptionEvent`
3. Consumer polls events, commits; `ConsensusSubscriptionCommitManager` tracks per-(group, topic, region) progress with contiguous-advance semantics for at-least-once delivery
4. WAL retention extended: safe-delete index = min(replication progress, subscription progress)
### New Files (5 core + 1 interface + 1 POJO)
- `ConsensusPrefetchingQueue` — dual-path (memory + WAL) prefetch queue, one per (group, topic, region)
- `ConsensusLogToTabletConverter` — InsertNode → Tablet conversion with Tree/Table pattern filtering
- `ConsensusSubscriptionCommitManager` — commit state persistence & recovery
- `ConsensusSubscriptionSetupHandler` — setup/teardown, auto-binds new regions via `IoTConsensus.onNewPeerCreated`
- `ConsensusSubscriptionBroker` — per-group broker managing multi-region queues
- `ISubscriptionBroker` — common interface for pipe/consensus brokers
- `SubscriptionConsensusProgress` — (searchIndex, commitIndex) POJO
### Modified Files
- `SubscriptionBrokerAgent` — dual broker routing (pipe + consensus)
- `IoTConsensusServerImpl` — subscription queue registration, WAL safe-delete adjustment
- `CreateSubscriptionProcedure` / `DropSubscriptionProcedure` — skip Pipe for consensus topics
- `SubscriptionConsumerAgent`, `ConsumerGroupMeta`, `SubscriptionBroker`, `SubscriptionEvent`
### Known Limitations
- No time-based filtering (WAL lacks time index)
- Seek not yet implemented
- Tests are manual (TODO: migrate to IT framework)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3108225897
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java:
##########
@@ -389,6 +389,13 @@ public class CommonConfig {
private long subscriptionMetaSyncerInitialSyncDelayMinutes = 3;
private long subscriptionMetaSyncerSyncIntervalMinutes = 3;
+ private int subscriptionConsensusBatchMaxDelayInMs = 50;
Review Comment:
All configs here support hot-reloading, but only affect newly created topics and won't affect existing topics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3278711861
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/WriterSafeFrontierTracker.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.consensus.iot;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Tracks per-writer safe frontier on the receiving side.
+ *
+ * <p>Each writer keeps at most one pending safeHLC because generated safeHLC for the same writer is
+ * expected to be totally ordered by both safePt and barrierLocalSeq.
+ */
+public class WriterSafeFrontierTracker {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(WriterSafeFrontierTracker.class);
+
+ private final Map<WriterIdentity, WriterFrontierState> states = new HashMap<>();
+
+ public synchronized void recordAppliedProgress(
+ final long physicalTime, final int writerNodeId, final long appliedLocalSeq) {
+ final WriterIdentity writerIdentity = new WriterIdentity(writerNodeId);
+ final WriterFrontierState state =
+ states.computeIfAbsent(writerIdentity, ignored -> new WriterFrontierState());
+ state.appliedLocalSeq = Math.max(state.appliedLocalSeq, appliedLocalSeq);
+ if (physicalTime > 0) {
+ state.effectiveSafePt = Math.max(state.effectiveSafePt, physicalTime);
+ }
+ promotePendingIfReady(state);
+ }
+
+ public synchronized void observePendingSafeHlc(
+ final long safePhysicalTime, final int writerNodeId, final long barrierLocalSeq) {
+ if (safePhysicalTime <= 0) {
+ return;
+ }
+ final WriterIdentity writerIdentity = new WriterIdentity(writerNodeId);
+ final WriterFrontierState state =
+ states.computeIfAbsent(writerIdentity, ignored -> new WriterFrontierState());
+ final SafeHlc candidate = new SafeHlc(safePhysicalTime, barrierLocalSeq);
+ if (state.appliedLocalSeq >= barrierLocalSeq) {
+ state.effectiveSafePt = Math.max(state.effectiveSafePt, safePhysicalTime);
+ state.pendingSafeHlc = null;
+ return;
+ }
+ if (state.pendingSafeHlc == null) {
+ state.pendingSafeHlc = candidate;
+ return;
+ }
+ final SafeHlc pending = state.pendingSafeHlc;
+ if (dominates(candidate, pending)) {
+ state.pendingSafeHlc = candidate;
+ return;
+ }
+ if (dominates(pending, candidate)) {
+ return;
+ }
+ LOGGER.warn(
+ "Observed incomparable safeHLC for writer {}. keep pending={}, ignore candidate={}",
+ writerIdentity,
+ pending,
+ candidate);
+ }
+
+ public synchronized long getEffectiveSafePt(final int writerNodeId) {
+ final WriterFrontierState state = states.get(new WriterIdentity(writerNodeId));
+ return Objects.nonNull(state) ? state.effectiveSafePt : 0L;
+ }
+
+ public synchronized SafeHlc getPendingSafeHlc(final int writerNodeId) {
+ final WriterFrontierState state = states.get(new WriterIdentity(writerNodeId));
+ return Objects.nonNull(state) ? state.pendingSafeHlc : null;
+ }
+
+ public synchronized Map<WriterIdentity, Long> snapshotEffectiveSafePts() {
+ final Map<WriterIdentity, Long> snapshot = new HashMap<>();
+ for (final Map.Entry<WriterIdentity, WriterFrontierState> entry : states.entrySet()) {
+ snapshot.put(entry.getKey(), entry.getValue().effectiveSafePt);
+ }
+ return Collections.unmodifiableMap(snapshot);
+ }
+
+ private void promotePendingIfReady(final WriterFrontierState state) {
+ if (state.pendingSafeHlc == null) {
+ return;
+ }
+ if (state.appliedLocalSeq >= state.pendingSafeHlc.getBarrierLocalSeq()) {
+ state.effectiveSafePt =
+ Math.max(state.effectiveSafePt, state.pendingSafeHlc.getSafePhysicalTime());
+ state.pendingSafeHlc = null;
+ }
+ }
+
+ private static boolean dominates(final SafeHlc left, final SafeHlc right) {
+ return left.safePhysicalTime >= right.safePhysicalTime
+ && left.barrierLocalSeq >= right.barrierLocalSeq;
+ }
+
+ public static final class WriterIdentity {
+ private final int writerNodeId;
+
+ public WriterIdentity(final int writerNodeId) {
+ this.writerNodeId = writerNodeId;
+ }
+
+ public int getWriterNodeId() {
+ return writerNodeId;
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (!(obj instanceof WriterIdentity)) {
+ return false;
+ }
+ final WriterIdentity that = (WriterIdentity) obj;
+ return writerNodeId == that.writerNodeId;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(writerNodeId);
+ }
+
+ @Override
+ public String toString() {
+ return "WriterIdentity{" + "writerNodeId=" + writerNodeId + '}';
+ }
+ }
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279186571
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -0,0 +1,1253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.common.rpc.thrift.TDataNodeLocation;
+import org.apache.iotdb.common.rpc.thrift.TEndPoint;
+import org.apache.iotdb.common.rpc.thrift.TRegionReplicaSet;
+import org.apache.iotdb.commons.client.ClientPoolFactory;
+import org.apache.iotdb.commons.client.IClientManager;
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient;
+import org.apache.iotdb.commons.consensus.ConfigRegionId;
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.confignode.rpc.thrift.TGetCommitProgressReq;
+import org.apache.iotdb.confignode.rpc.thrift.TGetCommitProgressResp;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.protocol.client.ConfigNodeClient;
+import org.apache.iotdb.db.protocol.client.ConfigNodeClientManager;
+import org.apache.iotdb.db.protocol.client.ConfigNodeInfo;
+import org.apache.iotdb.db.queryengine.plan.analyze.ClusterPartitionFetcher;
+import org.apache.iotdb.mpp.rpc.thrift.TSyncSubscriptionProgressReq;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Manages committed progress for consensus-based subscriptions.
+ *
+ * <p>State is maintained per {@code (consumerGroup, topic, region)} so each DataRegion can recover
+ * independently.
+ *
+ * <p>Committed progress is represented in per-writer terms via {@link WriterId} and {@link
+ * WriterProgress}. Outstanding deliveries are tracked by writer-local slots, while commit
+ * advancement is computed with ordered progress keys derived from {@code physicalTime}, {@code
+ * writerNodeId}, and {@code localSeq}. {@code searchIndex} is not the committed frontier here; it
+ * only remains an implementation aid for WAL positioning elsewhere.
+ *
+ * <p>Key responsibilities:
+ *
+ * <ul>
+ * <li>Track dispatched but uncommitted mappings per writer
+ * <li>Advance committed progress idempotently and contiguously on ack/commit
+ * <li>Persist, recover, and broadcast committed region progress
+ * </ul>
+ */
+public class ConsensusSubscriptionCommitManager {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(ConsensusSubscriptionCommitManager.class);
+
+ private static final String PROGRESS_FILE_PREFIX = "consensus_subscription_progress_";
+ private static final String PROGRESS_FILE_SUFFIX = ".dat";
+
+ private static final IClientManager<ConfigRegionId, ConfigNodeClient> CONFIG_NODE_CLIENT_MANAGER =
+ ConfigNodeClientManager.getInstance();
+
+ /** Client manager for DataNode-to-DataNode RPC (progress broadcast). */
+ private static final IClientManager<TEndPoint, SyncDataNodeInternalServiceClient>
+ SYNC_DN_CLIENT_MANAGER =
+ new IClientManager.Factory<TEndPoint, SyncDataNodeInternalServiceClient>()
+ .createClientManager(
+ new ClientPoolFactory.SyncDataNodeInternalServiceClientPoolFactory());
+
+ /** Minimum interval (ms) between broadcasts for the same (consumerGroup, topic, region). */
+ private static final long MIN_BROADCAST_INTERVAL_MS = 5000;
+
+ /** Rate-limiting: last broadcast timestamp per key. */
+ private final Map<String, Long> lastBroadcastTime = new ConcurrentHashMap<>();
+
+ /** Single-threaded executor for fire-and-forget broadcasts. */
+ private final ExecutorService broadcastExecutor =
+ Executors.newSingleThreadExecutor(
+ r -> {
+ final Thread t = new Thread(r, "SubscriptionProgressBroadcast");
+ t.setDaemon(true);
+ return t;
+ });
+
+ /** Key: "consumerGroupId##topicName##regionId" -> progress tracking state */
+ private final Map<String, ConsensusSubscriptionCommitState> commitStates =
+ new ConcurrentHashMap<>();
+
+ private final String persistDir;
+
+ private ConsensusSubscriptionCommitManager() {
+ this.persistDir =
+ IoTDBDescriptor.getInstance().getConfig().getSystemDir()
+ + File.separator
+ + "subscription"
+ + File.separator
+ + "consensus_progress";
+ final File dir = new File(persistDir);
+ if (!dir.exists()) {
+ dir.mkdirs();
+ }
+ }
+
+ /**
+ * Gets or creates the commit state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID
+ * @return the commit state
+ */
+ public ConsensusSubscriptionCommitState getOrCreateState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final String regionIdString = regionId.toString();
+ return commitStates.computeIfAbsent(
+ key,
+ k -> {
+ // Try to recover from persisted local state
+ final ConsensusSubscriptionCommitState recovered = tryRecover(key, regionIdString);
+ if (recovered != null) {
+ return recovered;
+ }
+ final ConsensusSubscriptionCommitState recoveredFromConfigNode =
+ queryCommitProgressStateFromConfigNode(consumerGroupId, topicName, regionId);
+ if (Objects.nonNull(recoveredFromConfigNode)) {
+ return recoveredFromConfigNode;
+ }
+ return new ConsensusSubscriptionCommitState(
+ regionIdString, new SubscriptionConsensusProgress());
+ });
+ }
+
+ public boolean hasPersistedState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ return getProgressFile(generateKey(consumerGroupId, topicName, regionId)).exists();
+ }
+
+ public void recordMapping(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final ConsensusSubscriptionCommitState state =
+ getOrCreateState(consumerGroupId, topicName, regionId);
+ state.recordMapping(writerId, writerProgress);
+ }
+
+ public boolean commit(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot commit for unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionId,
+ writerId,
+ writerProgress);
+ return false;
+ }
+ final CommitOperationResult result = state.commitAndGetResult(writerId, writerProgress);
+ if (result.isHandled()) {
+ // Periodically persist progress
+ persistProgressIfNeeded(key, state);
+ if (result.hasAdvancedWriter()) {
+ maybeBroadcast(
+ key,
+ consumerGroupId,
+ topicName,
+ regionId,
+ result.getAdvancedWriterProgress(),
+ result.getAdvancedWriterId());
+ }
+ }
+ return result.isHandled();
+ }
+
+ public boolean commitWithoutOutstanding(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot direct-commit for unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionId,
+ writerId,
+ writerProgress);
+ return false;
+ }
+ final CommitOperationResult result =
+ state.commitWithoutOutstandingAndGetResult(writerId, writerProgress);
+ if (result.isHandled()) {
+ persistProgressIfNeeded(key, state);
+ if (result.hasAdvancedWriter()) {
+ maybeBroadcast(
+ key,
+ consumerGroupId,
+ topicName,
+ regionId,
+ result.getAdvancedWriterProgress(),
+ result.getAdvancedWriterId());
+ }
+ }
+ return result.isHandled();
+ }
+
+ public long getCommittedPhysicalTime(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedPhysicalTime() : 0L;
+ }
+
+ public long getCommittedLocalSeq(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedLocalSeq() : -1L;
+ }
+
+ public int getCommittedWriterNodeId(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedWriterNodeId() : -1;
+ }
+
+ public WriterId getCommittedWriterId(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedWriterId() : null;
+ }
+
+ public WriterProgress getCommittedWriterProgress(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedWriterProgress() : null;
+ }
+
+ public RegionProgress getCommittedRegionProgress(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+ return state.getCommittedRegionProgress();
+ }
+
+ /**
+ * Removes state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID
+ */
+ public void removeState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ commitStates.remove(key);
+ // Clean up persisted file
+ final File file = getProgressFile(key);
+ if (file.exists()) {
+ file.delete();
+ }
+ }
+
+ /**
+ * Removes all states for a given (consumerGroup, topic) pair across all regions. Used during
+ * subscription teardown when the individual regionIds may not be readily available.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ */
+ public void removeAllStatesForTopic(final String consumerGroupId, final String topicName) {
+ final String prefix = consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR;
+ final Iterator<Map.Entry<String, ConsensusSubscriptionCommitState>> it =
+ commitStates.entrySet().iterator();
+ while (it.hasNext()) {
+ final Map.Entry<String, ConsensusSubscriptionCommitState> entry = it.next();
+ if (entry.getKey().startsWith(prefix)) {
+ it.remove();
+ final File file = getProgressFile(entry.getKey());
+ if (file.exists()) {
+ file.delete();
+ }
+ }
+ }
+ }
+
+ public void resetState(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final RegionProgress regionProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot reset unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}",
+ consumerGroupId,
+ topicName,
+ regionId);
+ return;
+ }
+ state.resetForSeek(regionProgress);
+ persistProgress(key, state);
+ }
+
+ /** Persists all states. Should be called during graceful shutdown. */
+ public void persistAll() {
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ commitStates.entrySet()) {
+ persistProgress(entry.getKey(), entry.getValue());
+ }
+ }
+
+ public Map<String, ByteBuffer> collectAllRegionProgress(final int dataNodeId) {
+ final Map<String, ByteBuffer> result = new ConcurrentHashMap<>();
+ final String suffix = KEY_SEPARATOR + dataNodeId;
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ commitStates.entrySet()) {
+ final RegionProgress regionProgress = entry.getValue().getCommittedRegionProgress();
+ final ByteBuffer serialized = serializeRegionProgress(regionProgress);
+ if (Objects.nonNull(serialized)) {
+ result.put(entry.getKey() + suffix, serialized);
+ }
+ }
+ return result;
+ }
+
+ // ======================== Progress Broadcast (Leader → Follower) ========================
+
+ /**
+ * Broadcasts committed progress to followers if enough time has elapsed since the last broadcast
+ * for this key. The broadcast is async and fire-and-forget.
+ */
+ private void maybeBroadcast(
+ final String key,
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterProgress committedWriterProgress,
+ final WriterId committedWriterId) {
+ if (Objects.isNull(committedWriterId) || Objects.isNull(committedWriterProgress)) {
+ return;
+ }
+ final String broadcastKey = buildBroadcastKey(key, committedWriterId);
+ final long now = System.currentTimeMillis();
+ final Long last = lastBroadcastTime.get(broadcastKey);
+ if (last != null && now - last < MIN_BROADCAST_INTERVAL_MS) {
+ return;
+ }
+ lastBroadcastTime.put(broadcastKey, now);
+ broadcastExecutor.submit(
+ () ->
+ doBroadcast(
+ consumerGroupId, topicName, regionId, committedWriterProgress, committedWriterId));
+ }
+
+ /**
+ * Sends committed progress to all follower replicas of the given region. Uses the partition cache
+ * to discover replica endpoints and skips the local DataNode.
+ */
+ private void doBroadcast(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterProgress writerProgress,
+ final WriterId writerId) {
+ final int localDataNodeId = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ try {
+ final List<TRegionReplicaSet> replicaSets =
+ ClusterPartitionFetcher.getInstance()
+ .getRegionReplicaSet(
+ Collections.singletonList(regionId.convertToTConsensusGroupId()));
+ if (replicaSets.isEmpty()) {
+ return;
+ }
+ final String regionIdStr = regionId.toString();
+ final int writerNodeId =
+ Objects.nonNull(writerId) && writerId.getNodeId() >= 0 ? writerId.getNodeId() : -1;
+ final TSyncSubscriptionProgressReq req =
+ new TSyncSubscriptionProgressReq(
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ Objects.nonNull(writerProgress) ? writerProgress.getPhysicalTime() : 0L,
+ writerNodeId,
+ Objects.nonNull(writerProgress) ? writerProgress.getLocalSeq() : -1L);
+
+ for (final TDataNodeLocation location : replicaSets.get(0).getDataNodeLocations()) {
+ if (location.getDataNodeId() == localDataNodeId) {
+ continue; // skip self
+ }
+ final TEndPoint endpoint = location.getInternalEndPoint();
+ try (final SyncDataNodeInternalServiceClient client =
+ SYNC_DN_CLIENT_MANAGER.borrowClient(endpoint)) {
+ client.syncSubscriptionProgress(req);
+ } catch (final ClientManagerException | TException e) {
+ LOGGER.debug(
+ "Failed to broadcast subscription progress to DataNode {} at {}: {}",
+ location.getDataNodeId(),
+ endpoint,
+ e.getMessage());
+ }
+ }
+ } catch (final Exception e) {
+ LOGGER.debug(
+ "Failed to broadcast subscription progress for region {}: {}", regionId, e.getMessage());
+ }
+ }
+
+ /**
+ * Receives a committed progress broadcast from another DataNode (Leader). Updates local state if
+ * the broadcast progress is ahead of the current local progress.
+ */
+ public void receiveProgressBroadcast(
+ final String consumerGroupId,
+ final String topicName,
+ final String regionIdStr,
+ final long physicalTime,
+ final int writerNodeId,
+ final long localSeq) {
+ receiveProgressBroadcast(
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ buildWriterId(regionIdStr, writerNodeId),
+ new WriterProgress(physicalTime, localSeq));
+ }
+
+ public void receiveProgressBroadcast(
+ final String consumerGroupId,
+ final String topicName,
+ final String regionIdStr,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: ignore broadcast without writer identity, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ writerId,
+ writerProgress);
+ return;
+ }
+ final String key = consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionIdStr;
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state != null) {
+ // Update only if broadcast is ahead
+ state.updateFromBroadcast(writerId, writerProgress);
+ persistProgressIfNeeded(key, state);
+ } else {
+ // Create a new state from the broadcast progress
+ final ConsensusSubscriptionCommitState newState =
+ new ConsensusSubscriptionCommitState(
+ regionIdStr,
+ new SubscriptionConsensusProgress(
+ new RegionProgress(Collections.singletonMap(writerId, writerProgress)), 0L));
+ newState.updateFromBroadcast(writerId, writerProgress);
+ commitStates.putIfAbsent(key, newState);
+ persistProgress(key, commitStates.get(key));
+ }
+ LOGGER.debug(
+ "Received subscription progress broadcast: consumerGroupId={}, topicName={}, "
+ + "regionId={}, physicalTime={}, localSeq={}",
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ writerProgress != null ? writerProgress.getPhysicalTime() : 0L,
+ writerProgress != null ? writerProgress.getLocalSeq() : -1L);
+ }
+
+ // ======================== Helper Methods ========================
+
+ // Use a separator that cannot appear in consumerGroupId, topicName, or regionId
+ // to prevent key collisions (e.g., "a_b" + "c" vs "a" + "b_c").
+ private static final String KEY_SEPARATOR = "##";
+
+ private String generateKey(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ return consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionId.toString();
Review Comment:
Added a test covering OS-forbidden characters plus recovery.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2928803499
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,1420 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+/**
+ * A prefetching queue that reads data from IoTConsensus using a hybrid approach:
+ *
+ * <ol>
+ * <li><b>In-memory pending queue</b>: Registered with {@link IoTConsensusServerImpl}, receives
+ * {@link IndexedConsensusRequest} in real-time from the write path (same mechanism as
+ * LogDispatcher). This avoids waiting for WAL flush to disk.
+ * <li><b>WAL fallback</b>: Uses {@link ConsensusReqReader.ReqIterator} to read from WAL files for
+ * gap-filling (pending queue overflow) or catch-up scenarios.
+ * <li><b>WAL pinning</b>: Supplies the earliest outstanding (uncommitted) search index to {@link
+ * IoTConsensusServerImpl}, preventing WAL deletion of entries not yet consumed by the
+ * subscription.
+ * </ol>
+ *
+ * <p>A background prefetch thread continuously drains the pending queue, converts InsertNode
+ * entries to Tablets via {@link ConsensusLogToTabletConverter}, and enqueues {@link
+ * SubscriptionEvent} objects into the prefetchingQueue for consumer polling.
+ *
+ * <p>This design mirrors LogDispatcher's dual-path (pendingEntries + WAL reader) but targets
+ * subscription delivery instead of replication.
+ *
+ * <p>Thread safety: Uses a fair {@link ReentrantReadWriteLock} to ensure mutual exclusion between
+ * cleanup and other operations (poll, ack, nack), consistent with the existing prefetching queue
+ * design.
+ */
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final String consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private volatile ConsensusReqReader.ReqIterator reqIterator;
+
+ /**
+ * In-memory pending queue registered with {@link IoTConsensusServerImpl#write}. Receives
+ * IndexedConsensusRequest in real-time without waiting for WAL flush. Capacity is bounded to
+ * apply back-pressure; overflows are filled from WAL.
+ */
+ private final BlockingQueue<IndexedConsensusRequest> pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ /** Commit ID generator, monotonically increasing within this queue's lifetime. */
+ private final AtomicLong commitIdGenerator;
+
+ /**
+ * Commit IDs less than or equal to this threshold are considered outdated. Updated on creation
+ * and on seek to invalidate all pre-seek events.
+ */
+ private volatile long outdatedCommitIdThreshold;
+
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ /**
+ * Tracks in-flight events that have been polled but not yet committed. Key: (consumerId,
+ * commitContext) -> event.
+ */
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ /**
+ * Tracks outstanding (uncommitted) events for WAL pinning. Maps commitId to the startSearchIndex
+ * of that event batch. The earliest entry's value is supplied to IoTConsensusServerImpl to pin
+ * WAL files from deletion.
+ */
+ private final ConcurrentSkipListMap<Long, Long> outstandingCommitIdToStartIndex;
Review Comment:
Why it is named as "outstanding"......
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionPullConsumer.java:
##########
@@ -195,6 +219,35 @@ protected List<SubscriptionMessage> poll(final Set<String> topicNames, final lon
return messages;
}
+ // Apply processor chain if configured
+ List<SubscriptionMessage> processed = messages;
+ if (!processors.isEmpty()) {
+ for (final SubscriptionMessageProcessor processor : processors) {
+ processed = processor.process(processed);
+ }
+ }
+
+ // Update watermark timestamp before stripping watermark events
+ for (final SubscriptionMessage m : processed) {
+ if (m.getMessageType() == SubscriptionMessageType.WATERMARK.getType()) {
+ final long ts = m.getWatermarkTimestamp();
+ if (ts > latestWatermarkTimestamp) {
+ latestWatermarkTimestamp = ts;
+ }
+ }
+ }
+
+ // Strip system messages — they are only for processors, not for users
+ processed.removeIf(
+ m -> {
+ final short type = m.getMessageType();
+ return type == SubscriptionMessageType.WATERMARK.getType();
+ });
+
+ if (processed.isEmpty()) {
+ return processed;
+ }
+
// add to uncommitted messages
Review Comment:
`SubscriptionMessageProcessor` can buffer messages locally (for example `WatermarkProcessor`), but only the post-
processor output is tracked/committed here. The server still considers the buffered events in-flight and can recycle/
re-deliver them after the uncommitted timeout. That means the same event can enter the processor more than once, which
breaks ordered-consumption correctness. We probably need a lease/heartbeat mechanism for processor-buffered events, or
another way to keep them pinned until the processor releases them.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionPullConsumer.java:
##########
@@ -135,6 +141,24 @@ public synchronized void close() {
return;
}
+ // flush all processors and commit any remaining buffered messages
+ if (!processors.isEmpty()) {
+ final List<SubscriptionMessage> flushed = new ArrayList<>();
+ for (final SubscriptionMessageProcessor processor : processors) {
+ final List<SubscriptionMessage> out = processor.flush();
+ if (out != null) {
+ flushed.addAll(out);
+ }
+ }
+ if (!flushed.isEmpty() && autoCommit) {
Review Comment:
On close we flush processor buffers, but we only commit the flushed messages when `autoCommit == true`. In manual-
commit mode those flushed messages are neither returned to the caller nor preserved for redelivery, so they can be
silently lost on close. We should either reject close with buffered processor state in manual mode, return the flushed
messages to the caller, or keep them recoverable.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/receiver/SubscriptionReceiverV1.java:
##########
@@ -669,22 +700,90 @@ private TPipeSubscribeResp handlePipeSubscribeCommitInternal(final PipeSubscribe
if (Objects.equals(successfulCommitContexts.size(), commitContexts.size())) {
LOGGER.info(
- "Subscription: consumer {} commit (nack: {}) successfully, commit contexts: {}",
+ "Subscription: consumer {} commit (nack: {}) successfully, summary: {}",
consumerConfig,
nack,
- commitContexts);
+ summarizeCommitContexts(commitContexts));
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug(
+ "Subscription: consumer {} commit (nack: {}) full commit contexts: {}",
+ consumerConfig,
+ nack,
+ commitContexts);
+ }
} else {
LOGGER.warn(
- "Subscription: consumer {} commit (nack: {}) partially successful, commit contexts: {}, successful commit contexts: {}",
+ "Subscription: consumer {} commit (nack: {}) partially successful, requested summary: {}, successful summary: {}",
consumerConfig,
nack,
- commitContexts,
- successfulCommitContexts);
+ summarizeCommitContexts(commitContexts),
+ summarizeCommitContexts(successfulCommitContexts));
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug(
+ "Subscription: consumer {} commit (nack: {}) full requested commit contexts: {}, full successful commit contexts: {}",
+ consumerConfig,
+ nack,
+ commitContexts,
+ successfulCommitContexts);
+ }
}
return PipeSubscribeCommitResp.toTPipeSubscribeResp(RpcUtils.SUCCESS_STATUS);
Review Comment:
`successfulCommitContexts` is already computed above, but the RPC still returns a generic success status even when
only part of the batch was actually committed. The client advances its local committed positions after any successful
RPC, so this can move the client checkpoint ahead of the real committed frontier and cause skipped data after
recovery/seek. We should either return the successful subset to the client or fail the RPC when the batch is only
partially committed.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -109,89 +201,319 @@ public List<SubscriptionCommitContext> commit(
final List<SubscriptionCommitContext> commitContexts,
final boolean nack) {
final String consumerGroupId = consumerConfig.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ final String consumerId = consumerConfig.getConsumerId();
+ final List<SubscriptionCommitContext> allSuccessful = new ArrayList<>();
+
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+
+ if (Objects.isNull(pipeBroker) && Objects.isNull(consensusBroker)) {
+ final String errorMessage =
+ String.format("Subscription: no broker bound to consumer group [%s]", consumerGroupId);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ // Partition commit contexts by which broker owns the topic.
+ final List<SubscriptionCommitContext> pipeContexts = new ArrayList<>();
+ final List<SubscriptionCommitContext> consensusContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext ctx : commitContexts) {
+ final String topicName = ctx.getTopicName();
+ if (Objects.nonNull(consensusBroker)
+ && ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName)) {
+ consensusContexts.add(ctx);
+ } else {
+ pipeContexts.add(ctx);
+ }
+ }
+
+ if (Objects.nonNull(pipeBroker) && !pipeContexts.isEmpty()) {
+ allSuccessful.addAll(pipeBroker.commit(consumerId, pipeContexts, nack));
+ }
+ if (Objects.nonNull(consensusBroker) && !consensusContexts.isEmpty()) {
+ allSuccessful.addAll(consensusBroker.commit(consumerId, consensusContexts, nack));
+ }
+
+ return allSuccessful;
+ }
+
+ public void seek(
+ final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
Review Comment:
`seek/seekAfter` currently requires every available provider to succeed, but on the server side a DataNode only
handles the request if it actually has a local consensus queue for this topic; otherwise it returns an error. In a
multi-DataNode deployment this makes the operation fail even when some providers simply do not own any relevant local
queue. The client/server contract needs to be aligned here: either route the request centrally, or treat non-owning
nodes as no-op success.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2870247778
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -109,89 +169,160 @@ public List<SubscriptionCommitContext> commit(
final List<SubscriptionCommitContext> commitContexts,
final boolean nack) {
final String consumerGroupId = consumerConfig.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ final String consumerId = consumerConfig.getConsumerId();
+ final List<SubscriptionCommitContext> allSuccessful = new ArrayList<>();
+
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+
+ if (Objects.isNull(pipeBroker) && Objects.isNull(consensusBroker)) {
final String errorMessage =
- String.format(
- "Subscription: broker bound to consumer group [%s] does not exist", consumerGroupId);
+ String.format("Subscription: no broker bound to consumer group [%s]", consumerGroupId);
LOGGER.warn(errorMessage);
throw new SubscriptionException(errorMessage);
}
- final String consumerId = consumerConfig.getConsumerId();
- return broker.commit(consumerId, commitContexts, nack);
+
+ // Partition commit contexts by which broker owns the topic.
+ final List<SubscriptionCommitContext> pipeContexts = new ArrayList<>();
+ final List<SubscriptionCommitContext> consensusContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext ctx : commitContexts) {
+ final String topicName = ctx.getTopicName();
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ consensusContexts.add(ctx);
+ } else {
+ pipeContexts.add(ctx);
+ }
+ }
+
+ if (Objects.nonNull(pipeBroker) && !pipeContexts.isEmpty()) {
+ allSuccessful.addAll(pipeBroker.commit(consumerId, pipeContexts, nack));
+ }
+ if (Objects.nonNull(consensusBroker) && !consensusContexts.isEmpty()) {
+ allSuccessful.addAll(consensusBroker.commit(consumerId, consensusContexts, nack));
+ }
+
+ return allSuccessful;
}
public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
final String consumerGroupId = commitContext.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ final String topicName = commitContext.getTopicName();
+
+ // Try consensus broker first
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ return consensusBroker.isCommitContextOutdated(commitContext);
+ }
+
+ // Fall back to pipe broker
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
return true;
}
- return broker.isCommitContextOutdated(commitContext);
+ return pipeBroker.isCommitContextOutdated(commitContext);
}
public List<String> fetchTopicNamesToUnsubscribe(
final ConsumerConfig consumerConfig, final Set<String> topicNames) {
final String consumerGroupId = consumerConfig.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+
+ // Consensus-based subscription topics are unbounded streams, so they do not trigger
+ // auto-unsubscribe.
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ final Set<String> pipeOnlyTopicNames;
+ if (Objects.nonNull(consensusBroker)) {
+ pipeOnlyTopicNames = new java.util.HashSet<>(topicNames);
+ pipeOnlyTopicNames.removeIf(consensusBroker::hasQueue);
+ } else {
+ pipeOnlyTopicNames = topicNames;
+ }
+
+ if (pipeOnlyTopicNames.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
return Collections.emptyList();
}
- return broker.fetchTopicNamesToUnsubscribe(topicNames);
+ return pipeBroker.fetchTopicNamesToUnsubscribe(pipeOnlyTopicNames);
}
/////////////////////////////// broker ///////////////////////////////
public boolean isBrokerExist(final String consumerGroupId) {
- return consumerGroupIdToSubscriptionBroker.containsKey(consumerGroupId);
+ return consumerGroupIdToPipeBroker.containsKey(consumerGroupId)
+ || consumerGroupIdToConsensusBroker.containsKey(consumerGroupId);
}
public void createBrokerIfNotExist(final String consumerGroupId) {
- consumerGroupIdToSubscriptionBroker.computeIfAbsent(consumerGroupId, SubscriptionBroker::new);
- LOGGER.info("Subscription: create broker bound to consumer group [{}]", consumerGroupId);
+ consumerGroupIdToPipeBroker.computeIfAbsent(consumerGroupId, SubscriptionBroker::new);
+ LOGGER.info("Subscription: create pipe broker bound to consumer group [{}]", consumerGroupId);
}
/**
* @return {@code true} if drop broker success, {@code false} otherwise
*/
public boolean dropBroker(final String consumerGroupId) {
final AtomicBoolean dropped = new AtomicBoolean(false);
- consumerGroupIdToSubscriptionBroker.compute(
+
+ // Drop pipe broker
+ consumerGroupIdToPipeBroker.compute(
consumerGroupId,
(id, broker) -> {
if (Objects.isNull(broker)) {
+ dropped.set(true);
Review Comment:
`dropBroker` sets `dropped=true` when the pipe broker entry is already null. This causes `dropBroker` to return true even when nothing was removed, and can also mask failure to drop a non-empty consensus broker (pipe broker null => dropped=true, consensus broker kept). Only mark `dropped` true when an existing broker entry is actually removed.
```suggestion
// No existing broker to drop
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.TreeSet;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Manages commit state for consensus-based subscriptions.
+ *
+ * <p>This manager tracks which events have been committed by consumers and maps commit IDs back to
+ * WAL search indices. It maintains the progress for each (consumerGroup, topic, region) triple and
+ * supports persistence and recovery.
+ *
+ * <p>Progress is tracked <b>per-region</b> because searchIndex is region-local — each DataRegion
+ * has its own independent WAL with its own searchIndex namespace. Using a single state per topic
+ * would cause TreeSet deduplication bugs when different regions emit the same searchIndex value.
+ *
+ * <p>Key responsibilities:
+ *
+ * <ul>
+ * <li>Track the mapping from commitId to searchIndex
+ * <li>Handle commit/ack from consumers
+ * <li>Persist and recover progress state
+ * </ul>
+ */
+public class ConsensusSubscriptionCommitManager {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(ConsensusSubscriptionCommitManager.class);
+
+ private static final String PROGRESS_FILE_PREFIX = "consensus_subscription_progress_";
+ private static final String PROGRESS_FILE_SUFFIX = ".dat";
+
+ /** Key: "consumerGroupId_topicName_regionId" -> progress tracking state */
+ private final Map<String, ConsensusSubscriptionCommitState> commitStates =
+ new ConcurrentHashMap<>();
+
+ private final String persistDir;
+
+ private ConsensusSubscriptionCommitManager() {
+ this.persistDir =
+ IoTDBDescriptor.getInstance().getConfig().getSystemDir()
+ + File.separator
+ + "subscription"
+ + File.separator
+ + "consensus_progress";
+ final File dir = new File(persistDir);
+ if (!dir.exists()) {
+ dir.mkdirs();
+ }
+ }
+
+ /**
+ * Gets or creates the commit state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID string
+ * @return the commit state
+ */
+ public ConsensusSubscriptionCommitState getOrCreateState(
+ final String consumerGroupId, final String topicName, final String regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ return commitStates.computeIfAbsent(
+ key,
+ k -> {
+ // Try to recover from persisted state
+ final ConsensusSubscriptionCommitState recovered = tryRecover(key);
+ if (recovered != null) {
+ return recovered;
+ }
+ return new ConsensusSubscriptionCommitState(new SubscriptionConsensusProgress(0L, 0L));
+ });
+ }
+
+ /**
+ * Records commitId to searchIndex mapping for later commit handling.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID string
+ * @param commitId the assigned commit ID
+ * @param searchIndex the WAL search index corresponding to this event
+ */
+ public void recordCommitMapping(
+ final String consumerGroupId,
+ final String topicName,
+ final String regionId,
+ final long commitId,
+ final long searchIndex) {
+ final ConsensusSubscriptionCommitState state =
+ getOrCreateState(consumerGroupId, topicName, regionId);
+ state.recordMapping(commitId, searchIndex);
+ }
+
+ /**
+ * Handles commit (ack) for an event. Updates the progress and potentially advances the committed
+ * search index.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID string
+ * @param commitId the committed event's commit ID
+ * @return true if commit handled successfully
+ */
+ public boolean commit(
+ final String consumerGroupId,
+ final String topicName,
+ final String regionId,
+ final long commitId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot commit for unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}, commitId={}",
+ consumerGroupId,
+ topicName,
+ regionId,
+ commitId);
+ return false;
+ }
+ final boolean success = state.commit(commitId);
+ if (success) {
+ // Periodically persist progress
+ persistProgressIfNeeded(key, state);
+ }
+ return success;
+ }
+
+ /**
+ * Gets the current committed search index for a specific region's state.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID string
+ * @return the committed search index, or -1 if no state exists
+ */
+ public long getCommittedSearchIndex(
+ final String consumerGroupId, final String topicName, final String regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ return -1;
+ }
+ return state.getCommittedSearchIndex();
+ }
+
+ /**
+ * Removes state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID string
+ */
+ public void removeState(
+ final String consumerGroupId, final String topicName, final String regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ commitStates.remove(key);
+ // Clean up persisted file
+ final File file = getProgressFile(key);
+ if (file.exists()) {
+ file.delete();
+ }
+ }
+
+ /**
+ * Removes all states for a given (consumerGroup, topic) pair across all regions. Used during
+ * subscription teardown when the individual regionIds may not be readily available.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ */
+ public void removeAllStatesForTopic(final String consumerGroupId, final String topicName) {
+ final String prefix = consumerGroupId + "_" + topicName + "_";
+ final Iterator<Map.Entry<String, ConsensusSubscriptionCommitState>> it =
+ commitStates.entrySet().iterator();
+ while (it.hasNext()) {
+ final Map.Entry<String, ConsensusSubscriptionCommitState> entry = it.next();
+ if (entry.getKey().startsWith(prefix)) {
+ it.remove();
+ final File file = getProgressFile(entry.getKey());
+ if (file.exists()) {
+ file.delete();
+ }
+ }
+ }
+ }
+
+ /** Persists all states. Should be called during graceful shutdown. */
+ public void persistAll() {
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ commitStates.entrySet()) {
+ persistProgress(entry.getKey(), entry.getValue());
+ }
+ }
+
+ // ======================== Helper Methods ========================
+
+ private String generateKey(
+ final String consumerGroupId, final String topicName, final String regionId) {
+ return consumerGroupId + "_" + topicName + "_" + regionId;
+ }
+
+ private File getProgressFile(final String key) {
+ return new File(persistDir, PROGRESS_FILE_PREFIX + key + PROGRESS_FILE_SUFFIX);
+ }
+
+ private ConsensusSubscriptionCommitState tryRecover(final String key) {
+ final File file = getProgressFile(key);
+ if (!file.exists()) {
+ return null;
+ }
+ try (final FileInputStream fis = new FileInputStream(file)) {
+ final byte[] bytes = new byte[(int) file.length()];
+ fis.read(bytes);
Review Comment:
`tryRecover` reads the progress file with a single `fis.read(bytes)` call, which is not guaranteed to fill the buffer. This can lead to partial reads and corrupted/failed deserialization. Use a readFully loop / `DataInputStream.readFully`, and validate that the expected number of bytes was read before deserializing.
```suggestion
int offset = 0;
while (offset < bytes.length) {
final int bytesRead = fis.read(bytes, offset, bytes.length - offset);
if (bytesRead < 0) {
break;
}
offset += bytesRead;
}
if (offset != bytes.length) {
LOGGER.warn(
"Failed to fully read consensus subscription progress from {}: expected {} bytes, read {} bytes",
file,
bytes.length,
offset);
return null;
}
```
##########
example/session/pom.xml:
##########
@@ -40,4 +40,17 @@
<version>${project.version}</version>
</dependency>
</dependencies>
+ <!-- TODO: remove below -->
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <source>11</source>
+ <target>11</target>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
Review Comment:
This module overrides the example parent’s compiler level (example/pom.xml sets source/target=8) to Java 11. That can break building examples under the project’s expected Java version and makes the module inconsistent with other examples. Prefer inheriting the parent compiler settings (or updating the parent if the whole examples tree should move to 11) instead of overriding in this module.
```suggestion
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionSetupHandler.java:
##########
@@ -0,0 +1,422 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.IoTDBTreePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.PrefixTreePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.consensus.IConsensus;
+import org.apache.iotdb.consensus.iot.IoTConsensus;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.consensus.DataRegionConsensusImpl;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.subscription.agent.SubscriptionAgent;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Handles the setup and teardown of consensus-based subscription queues on DataNode. When a
+ * real-time subscription is detected, this handler finds the local IoTConsensus data regions,
+ * creates the appropriate converter, and binds prefetching queues to the subscription broker.
+ */
+public class ConsensusSubscriptionSetupHandler {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(ConsensusSubscriptionSetupHandler.class);
+
+ private static final IoTDBConfig IOTDB_CONFIG = IoTDBDescriptor.getInstance().getConfig();
+
+ private ConsensusSubscriptionSetupHandler() {
+ // utility class
+ }
+
+ /**
+ * Ensures that the IoTConsensus new-peer callback is set, so that when a new DataRegion is
+ * created, all active consensus subscriptions are automatically bound to the new region.
+ */
+ public static void ensureNewRegionListenerRegistered() {
+ if (IoTConsensus.onNewPeerCreated != null) {
+ return;
+ }
+ IoTConsensus.onNewPeerCreated = ConsensusSubscriptionSetupHandler::onNewRegionCreated;
+ LOGGER.info(
+ "Set IoTConsensus.onNewPeerCreated callback for consensus subscription auto-binding");
+ }
+
+ /**
+ * Callback invoked when a new DataRegion (IoTConsensusServerImpl) is created locally. Queries
+ * existing subscription metadata to find all active consensus subscriptions and binds prefetching
+ * queues to the new region.
+ */
+ private static void onNewRegionCreated(
+ final ConsensusGroupId groupId, final IoTConsensusServerImpl serverImpl) {
+ if (!(groupId instanceof DataRegionId)) {
+ return;
+ }
+
+ // Query existing metadata keepers for all active subscriptions
+ final Map<String, java.util.Set<String>> allSubscriptions =
+ SubscriptionAgent.consumer().getAllSubscriptions();
+ if (allSubscriptions.isEmpty()) {
+ return;
+ }
+
+ final ConsensusSubscriptionCommitManager commitManager =
+ ConsensusSubscriptionCommitManager.getInstance();
+ final long startSearchIndex = serverImpl.getSearchIndex() + 1;
+
+ LOGGER.info(
+ "New DataRegion {} created, checking {} consumer group(s) for auto-binding, "
+ + "startSearchIndex={}",
+ groupId,
+ allSubscriptions.size(),
+ startSearchIndex);
+
+ for (final Map.Entry<String, java.util.Set<String>> groupEntry : allSubscriptions.entrySet()) {
+ final String consumerGroupId = groupEntry.getKey();
+ for (final String topicName : groupEntry.getValue()) {
+ if (!isConsensusBasedTopic(topicName)) {
+ continue;
+ }
+ try {
+ final Map<String, TopicConfig> topicConfigs =
+ SubscriptionAgent.topic().getTopicConfigs(java.util.Collections.singleton(topicName));
+ final TopicConfig topicConfig = topicConfigs.get(topicName);
+ if (topicConfig == null) {
+ continue;
+ }
+
+ // Resolve the new DataRegion's actual database name
+ final DataRegion dataRegion =
+ StorageEngine.getInstance().getDataRegion((DataRegionId) groupId);
+ if (dataRegion == null) {
+ continue;
+ }
+ final String dbRaw = dataRegion.getDatabaseName();
+ final String dbTableModel = dbRaw.startsWith("root.") ? dbRaw.substring(5) : dbRaw;
+
+ // For table topics, skip if this region's database doesn't match the topic filter
+ if (topicConfig.isTableTopic()) {
+ final String topicDb =
+ topicConfig.getStringOrDefault(
+ TopicConstant.DATABASE_KEY, TopicConstant.DATABASE_DEFAULT_VALUE);
+ if (topicDb != null
+ && !topicDb.isEmpty()
+ && !TopicConstant.DATABASE_DEFAULT_VALUE.equals(topicDb)
+ && !topicDb.equalsIgnoreCase(dbTableModel)) {
+ continue;
+ }
+ }
+
+ final String actualDbName = topicConfig.isTableTopic() ? dbTableModel : null;
+ final ConsensusLogToTabletConverter converter = buildConverter(topicConfig, actualDbName);
+
+ LOGGER.info(
+ "Auto-binding consensus queue for topic [{}] in group [{}] to new region {} (database={})",
+ topicName,
+ consumerGroupId,
+ groupId,
+ dbTableModel);
+
+ SubscriptionAgent.broker()
+ .bindConsensusPrefetchingQueue(
+ consumerGroupId,
+ topicName,
+ groupId.toString(),
+ serverImpl,
+ converter,
+ commitManager,
+ startSearchIndex);
+ } catch (final Exception e) {
+ LOGGER.error(
+ "Failed to auto-bind topic [{}] in group [{}] to new region {}",
+ topicName,
+ consumerGroupId,
+ groupId,
+ e);
+ }
+ }
+ }
+ }
+
+ public static boolean isConsensusBasedTopic(final String topicName) {
+ try {
+ final String topicMode = SubscriptionAgent.topic().getTopicMode(topicName);
+ final String topicFormat = SubscriptionAgent.topic().getTopicFormat(topicName);
+ final boolean result =
+ TopicConstant.MODE_LIVE_VALUE.equalsIgnoreCase(topicMode)
+ && !TopicConstant.FORMAT_TS_FILE_HANDLER_VALUE.equalsIgnoreCase(topicFormat);
+ LOGGER.info(
+ "isConsensusBasedTopic check for topic [{}]: mode={}, format={}, result={}",
+ topicName,
+ topicMode,
+ topicFormat,
+ result);
+ return result;
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "Failed to check if topic [{}] is consensus-based, defaulting to false", topicName, e);
+ return false;
+ }
+ }
+
+ public static void setupConsensusSubscriptions(
+ final String consumerGroupId, final Set<String> topicNames) {
+ final IConsensus dataRegionConsensus = DataRegionConsensusImpl.getInstance();
+ if (!(dataRegionConsensus instanceof IoTConsensus)) {
+ LOGGER.warn(
+ "Data region consensus is not IoTConsensus (actual: {}), "
+ + "cannot set up consensus-based subscription for consumer group [{}]",
+ dataRegionConsensus.getClass().getSimpleName(),
+ consumerGroupId);
+ return;
+ }
+
+ // Ensure the new-region listener is registered (idempotent)
+ ensureNewRegionListenerRegistered();
+
+ final IoTConsensus ioTConsensus = (IoTConsensus) dataRegionConsensus;
+ final ConsensusSubscriptionCommitManager commitManager =
+ ConsensusSubscriptionCommitManager.getInstance();
+
+ LOGGER.info(
+ "Setting up consensus subscriptions for consumer group [{}], topics={}, "
+ + "total consensus groups={}",
+ consumerGroupId,
+ topicNames,
+ ioTConsensus.getAllConsensusGroupIds().size());
+
+ for (final String topicName : topicNames) {
+ if (!isConsensusBasedTopic(topicName)) {
+ continue;
+ }
+
+ try {
+ setupConsensusQueueForTopic(consumerGroupId, topicName, ioTConsensus, commitManager);
+ } catch (final Exception e) {
+ LOGGER.error(
+ "Failed to set up consensus subscription for topic [{}] in consumer group [{}]",
+ topicName,
+ consumerGroupId,
+ e);
+ }
+ }
+ }
+
+ /**
+ * Set up consensus queue for a single topic. Discovers all local data region consensus groups and
+ * binds a ConsensusReqReader-based prefetching queue to every matching region.
+ *
+ * <p>For table-model topics, only regions whose database matches the topic's {@code DATABASE_KEY}
+ * filter are bound. For tree-model topics, all local data regions are bound. Additionally, the
+ * {@link #onNewRegionCreated} callback ensures that regions created after this method runs are
+ * also automatically bound.
+ */
+ private static void setupConsensusQueueForTopic(
+ final String consumerGroupId,
+ final String topicName,
+ final IoTConsensus ioTConsensus,
+ final ConsensusSubscriptionCommitManager commitManager) {
+
+ // Get topic config for building the converter
+ final Map<String, TopicConfig> topicConfigs =
+ SubscriptionAgent.topic().getTopicConfigs(java.util.Collections.singleton(topicName));
+ final TopicConfig topicConfig = topicConfigs.get(topicName);
+ if (topicConfig == null) {
+ LOGGER.warn(
+ "Topic config not found for topic [{}], cannot set up consensus queue", topicName);
+ return;
+ }
+
+ // Build the converter based on topic config (path pattern, time range, tree/table model)
+ LOGGER.info(
+ "Setting up consensus queue for topic [{}]: isTableTopic={}, config={}",
+ topicName,
+ topicConfig.isTableTopic(),
+ topicConfig.getAttribute());
+
+ // For table topics, extract the database filter from topic config
+ final String topicDatabaseFilter =
+ topicConfig.isTableTopic()
+ ? topicConfig.getStringOrDefault(
+ TopicConstant.DATABASE_KEY, TopicConstant.DATABASE_DEFAULT_VALUE)
+ : null;
+
+ final List<ConsensusGroupId> allGroupIds = ioTConsensus.getAllConsensusGroupIds();
+ LOGGER.info(
+ "Discovered {} consensus group(s) for topic [{}] in consumer group [{}]: {}",
+ allGroupIds.size(),
+ topicName,
+ consumerGroupId,
+ allGroupIds);
+ boolean bound = false;
+
+ for (final ConsensusGroupId groupId : allGroupIds) {
+ if (!(groupId instanceof DataRegionId)) {
+ continue;
+ }
+
+ final IoTConsensusServerImpl serverImpl = ioTConsensus.getImpl(groupId);
+ if (serverImpl == null) {
+ continue;
+ }
+
+ // Resolve the DataRegion's actual database name
+ final DataRegion dataRegion =
+ StorageEngine.getInstance().getDataRegion((DataRegionId) groupId);
+ if (dataRegion == null) {
+ continue;
+ }
+ final String dbRaw = dataRegion.getDatabaseName();
+ final String dbTableModel = dbRaw.startsWith("root.") ? dbRaw.substring(5) : dbRaw;
+
+ if (topicDatabaseFilter != null
+ && !topicDatabaseFilter.isEmpty()
+ && !TopicConstant.DATABASE_DEFAULT_VALUE.equals(topicDatabaseFilter)
+ && !topicDatabaseFilter.equalsIgnoreCase(dbTableModel)) {
+ LOGGER.info(
+ "Skipping region {} (database={}) for table topic [{}] (DATABASE_KEY={})",
+ groupId,
+ dbTableModel,
+ topicName,
+ topicDatabaseFilter);
+ continue;
+ }
+
+ final String actualDbName = topicConfig.isTableTopic() ? dbTableModel : null;
+ final ConsensusLogToTabletConverter converter = buildConverter(topicConfig, actualDbName);
+
+ final long startSearchIndex = serverImpl.getSearchIndex() + 1;
+
Review Comment:
`startSearchIndex` is always initialized as `serverImpl.getSearchIndex() + 1`, which ignores any persisted subscription progress in `ConsensusSubscriptionCommitManager`. After a DataNode restart (or queue rebind), this will skip unconsumed WAL entries and break the at-least-once / recovery semantics described in the PR. The start index should be derived from the recovered committed search index when state exists (e.g., `max(committed+1, subscribeStart)` depending on desired semantics).
```suggestion
long startSearchIndex = serverImpl.getSearchIndex() + 1;
// If we have recovered a committed search index for this group, advance the
// startSearchIndex to at least committed + 1 to honor at-least-once semantics.
if (commitManager != null) {
try {
final long committedSearchIndex = commitManager.getCommittedSearchIndex(groupId);
if (committedSearchIndex >= 0) {
startSearchIndex = Math.max(startSearchIndex, committedSearchIndex + 1);
}
} catch (Exception e) {
LOGGER.warn(
"Failed to load committed search index for consensus group {} when binding "
+ "subscription prefetching queue, fallback to startSearchIndex={}",
groupId,
startSearchIndex,
e);
}
}
```
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/subscription/config/SubscriptionConfig.java:
##########
@@ -30,7 +30,7 @@ public class SubscriptionConfig {
private static final CommonConfig COMMON_CONFIG = CommonDescriptor.getInstance().getConfig();
public boolean getSubscriptionEnabled() {
- return false;
+ return true; // TODO: make it configurable after subscription is stable
Review Comment:
`getSubscriptionEnabled()` is now hard-coded to always return true, which changes the default behavior globally (multiple code paths gate subscription operations on this flag). This should remain configurable (or default to the previous disabled state) to avoid unexpectedly enabling an unstable feature in production deployments.
```suggestion
return false; // TODO: make it configurable or enable by default after subscription is stable
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/ConsensusSubscriptionBroker.java:
##########
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker;
+
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusLogToTabletConverter;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusPrefetchingQueue;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusSubscriptionCommitManager;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * Consensus-based subscription broker that reads data directly from IoTConsensus WAL. Each instance
+ * manages consensus prefetching queues for a single consumer group.
+ */
+public class ConsensusSubscriptionBroker implements ISubscriptionBroker {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusSubscriptionBroker.class);
+
+ private final String brokerId; // consumer group id
+
+ /** Maps topic name to a list of ConsensusPrefetchingQueues, one per data region. */
+ private final Map<String, List<ConsensusPrefetchingQueue>> topicNameToConsensusPrefetchingQueues;
+
+ /** Shared commit ID generators per topic. */
+ private final Map<String, AtomicLong> topicNameToCommitIdGenerator;
+
+ public ConsensusSubscriptionBroker(final String brokerId) {
+ this.brokerId = brokerId;
+ this.topicNameToConsensusPrefetchingQueues = new ConcurrentHashMap<>();
+ this.topicNameToCommitIdGenerator = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return topicNameToConsensusPrefetchingQueues.isEmpty();
+ }
+
+ @Override
+ public boolean hasQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ return Objects.nonNull(queues)
+ && !queues.isEmpty()
+ && queues.stream().anyMatch(q -> !q.isClosed());
+ }
+
+ //////////////////////////// poll ////////////////////////////
+
+ @Override
+ public List<SubscriptionEvent> poll(
+ final String consumerId, final Set<String> topicNames, final long maxBytes) {
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll called, consumerId={}, topicNames={}, "
+ + "queueCount={}, maxBytes={}",
+ brokerId,
+ consumerId,
+ topicNames,
+ topicNameToConsensusPrefetchingQueues.size(),
+ maxBytes);
+
+ final List<SubscriptionEvent> eventsToPoll = new ArrayList<>();
+ final List<SubscriptionEvent> eventsToNack = new ArrayList<>();
+ long totalSize = 0;
+
+ for (final String topicName : topicNames) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ continue;
+ }
+
+ // Poll from all region queues for this topic
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+
+ final SubscriptionEvent event = consensusQueue.poll(consumerId);
+ if (Objects.isNull(event)) {
+ continue;
+ }
+
+ final long currentSize;
+ try {
+ currentSize = event.getCurrentResponseSize();
+ } catch (final IOException e) {
+ eventsToNack.add(event);
+ continue;
+ }
+
+ eventsToPoll.add(event);
+ totalSize += currentSize;
+
+ if (totalSize + currentSize > maxBytes) {
Review Comment:
The `maxBytes` limiting logic is incorrect: the event is added and `totalSize` is incremented, then the code checks `if (totalSize + currentSize > maxBytes)` which double-counts `currentSize` and can still return a batch exceeding `maxBytes`. Check the size *before* adding the event (or adjust the condition) and decide whether to stop/push back the event when it would exceed the limit.
```suggestion
if (totalSize > maxBytes) {
```
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java:
##########
@@ -879,10 +972,25 @@ void checkAndUpdateSafeDeletedSearchIndex() {
if (configuration.isEmpty()) {
logger.error(
"Configuration is empty, which is unexpected. Safe deleted search index won't be updated this time.");
- } else if (configuration.size() == 1) {
+ return;
+ }
+
+ // Compute the minimum search index that subscription consumers still need.
+ // WAL entries at or after this index must be preserved.
+ long minSubscriptionIndex = Long.MAX_VALUE;
+ for (final LongSupplier supplier : subscriptionSyncIndexSuppliers) {
+ minSubscriptionIndex = Math.min(minSubscriptionIndex, supplier.getAsLong());
+ }
+
+ if (configuration.size() == 1 && subscriptionSyncIndexSuppliers.isEmpty()) {
+ // Single replica, no subscription consumers => delete all WAL freely
consensusReqReader.setSafelyDeletedSearchIndex(Long.MAX_VALUE);
} else {
- consensusReqReader.setSafelyDeletedSearchIndex(getMinFlushedSyncIndex());
+ // min(replication progress, subscription progress) — preserve WAL for both
+ final long replicationIndex =
+ configuration.size() > 1 ? getMinFlushedSyncIndex() : Long.MAX_VALUE;
+ consensusReqReader.setSafelyDeletedSearchIndex(
+ Math.min(replicationIndex, minSubscriptionIndex));
Review Comment:
`checkAndUpdateSafeDeletedSearchIndex()` now computes a subscription-aware safe delete index, but other code paths (e.g., `LogDispatcher` periodically calling `reader.setSafelyDeletedSearchIndex(impl.getMinFlushedSyncIndex())`) can overwrite it with a replication-only value. That can allow WAL deletion past the slowest subscription consumer, causing data loss. Consider centralizing *all* updates to safelyDeletedSearchIndex through a single subscription-aware method (or make `getMinFlushedSyncIndex()` incorporate subscription progress).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279128688
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -109,89 +201,319 @@ public List<SubscriptionCommitContext> commit(
final List<SubscriptionCommitContext> commitContexts,
final boolean nack) {
final String consumerGroupId = consumerConfig.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ final String consumerId = consumerConfig.getConsumerId();
+ final List<SubscriptionCommitContext> allSuccessful = new ArrayList<>();
+
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+
+ if (Objects.isNull(pipeBroker) && Objects.isNull(consensusBroker)) {
+ final String errorMessage =
+ String.format("Subscription: no broker bound to consumer group [%s]", consumerGroupId);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ // Partition commit contexts by which broker owns the topic.
+ final List<SubscriptionCommitContext> pipeContexts = new ArrayList<>();
+ final List<SubscriptionCommitContext> consensusContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext ctx : commitContexts) {
+ final String topicName = ctx.getTopicName();
+ if (Objects.nonNull(consensusBroker)
+ && ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName)) {
+ consensusContexts.add(ctx);
+ } else {
+ pipeContexts.add(ctx);
+ }
+ }
+
+ if (Objects.nonNull(pipeBroker) && !pipeContexts.isEmpty()) {
+ allSuccessful.addAll(pipeBroker.commit(consumerId, pipeContexts, nack));
+ }
+ if (Objects.nonNull(consensusBroker) && !consensusContexts.isEmpty()) {
+ allSuccessful.addAll(consensusBroker.commit(consumerId, consensusContexts, nack));
+ }
+
+ return allSuccessful;
+ }
+
+ public void seek(
+ final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ ensureConsensusSeekRuntimeAvailable(consumerGroupId, topicName, "seek");
+ if (seekType != PipeSubscribeSeekReq.SEEK_TO_BEGINNING
+ && seekType != PipeSubscribeSeekReq.SEEK_TO_END) {
+ final String errorMessage =
+ String.format(
+ "Subscription: consensus seek only supports beginning/end or topic progress, "
+ + "consumerGroup=%s, topic=%s, seekType=%s",
+ consumerGroupId, topicName, seekType);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+ consensusBroker.seek(topicName, seekType);
+ return;
+ }
+
+ if (isConsensusRuntimeUnsupported(topicName)) {
+ final String errorMessage =
+ buildUnsupportedConsensusRuntimeMessage(consumerGroupId, topicName, "seek");
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ final String errorMessage =
+ String.format(
+ "Subscription: seek is only supported for consensus-based subscriptions, "
+ + "consumerGroup=%s, topic=%s",
+ consumerGroupId, topicName);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ public void seekToTopicProgress(
+ final ConsumerConfig consumerConfig,
+ final String topicName,
+ final TopicProgress topicProgress) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ ensureConsensusSeekRuntimeAvailable(consumerGroupId, topicName, "seek(topicProgress)");
+ consensusBroker.seek(topicName, topicProgress);
+ return;
+ }
+
+ if (isConsensusRuntimeUnsupported(topicName)) {
+ final String errorMessage =
+ buildUnsupportedConsensusRuntimeMessage(
+ consumerGroupId, topicName, "seek(topicProgress)");
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ final String errorMessage =
+ String.format(
+ "Subscription: seek(topicProgress) is only supported for consensus-based subscriptions, "
+ + "consumerGroup=%s, topic=%s",
+ consumerGroupId, topicName);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ public void seekAfterTopicProgress(
+ final ConsumerConfig consumerConfig,
+ final String topicName,
+ final TopicProgress topicProgress) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ ensureConsensusSeekRuntimeAvailable(consumerGroupId, topicName, "seekAfter(topicProgress)");
+ consensusBroker.seekAfter(topicName, topicProgress);
+ return;
+ }
+
+ if (isConsensusRuntimeUnsupported(topicName)) {
+ final String errorMessage =
+ buildUnsupportedConsensusRuntimeMessage(
+ consumerGroupId, topicName, "seekAfter(topicProgress)");
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ final String errorMessage =
+ String.format(
+ "Subscription: seekAfter(topicProgress) is only supported for consensus-based subscriptions, "
+ + "consumerGroup=%s, topic=%s",
+ consumerGroupId, topicName);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ private void ensureConsensusSeekRuntimeAvailable(
+ final String consumerGroupId, final String topicName, final String operation) {
+ if (!ConsensusSubscriptionPrefetchExecutorManager.getInstance().isStarted()
+ || SubscriptionAgent.runtime().isShutdown()) {
final String errorMessage =
String.format(
- "Subscription: broker bound to consumer group [%s] does not exist", consumerGroupId);
+ "Subscription: consensus %s is unavailable because subscription runtime is stopped, "
+ + "consumerGroup=%s, topic=%s",
+ operation, consumerGroupId, topicName);
LOGGER.warn(errorMessage);
throw new SubscriptionException(errorMessage);
}
- final String consumerId = consumerConfig.getConsumerId();
- return broker.commit(consumerId, commitContexts, nack);
+ }
+
+ private boolean isConsensusRuntimeUnsupported(final String topicName) {
+ return !(DataRegionConsensusImpl.getInstance() instanceof IoTConsensus)
+ && ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName);
+ }
+
+ private List<String> getUnsupportedConsensusTopics(final Set<String> topicNames) {
+ if (DataRegionConsensusImpl.getInstance() instanceof IoTConsensus) {
+ return Collections.emptyList();
+ }
+
+ final List<String> unsupportedConsensusTopics = new ArrayList<>();
+ for (final String topicName : topicNames) {
+ if (ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName)) {
+ unsupportedConsensusTopics.add(topicName);
+ }
+ }
+ return unsupportedConsensusTopics;
+ }
+
+ private String buildUnsupportedConsensusRuntimeMessage(
+ final String consumerGroupId, final String topicName, final String operation) {
+ return buildUnsupportedConsensusRuntimeMessage(
+ consumerGroupId, Collections.singletonList(topicName), operation);
+ }
+
+ private String buildUnsupportedConsensusRuntimeMessage(
+ final String consumerGroupId, final List<String> topicNames, final String operation) {
+ final IConsensus dataRegionConsensus = DataRegionConsensusImpl.getInstance();
+ final String configuredProtocol =
+ IoTDBDescriptor.getInstance().getConfig().getDataRegionConsensusProtocolClass();
+ final String runtimeConsensusImplementation =
+ Objects.nonNull(dataRegionConsensus) ? dataRegionConsensus.getClass().getName() : "null";
+ return String.format(
+ "Subscription: cannot %s consensus-based topic(s) %s in consumer group [%s] because "
+ + "mode=consensus only supports data_region_consensus_protocol_class=%s, but current "
+ + "configured value is %s (runtime consensus implementation: %s)",
+ operation,
+ topicNames,
+ consumerGroupId,
+ ConsensusFactory.IOT_CONSENSUS,
+ configuredProtocol,
+ runtimeConsensusImplementation);
}
public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
final String consumerGroupId = commitContext.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ final String topicName = commitContext.getTopicName();
+
+ // Try consensus broker first
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ return consensusBroker.isCommitContextOutdated(commitContext);
+ }
+
+ // Fall back to pipe broker
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
return true;
}
- return broker.isCommitContextOutdated(commitContext);
+ return pipeBroker.isCommitContextOutdated(commitContext);
}
public List<String> fetchTopicNamesToUnsubscribe(
final ConsumerConfig consumerConfig, final Set<String> topicNames) {
final String consumerGroupId = consumerConfig.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+
+ // Consensus-based subscription topics are unbounded streams, so they do not trigger
+ // auto-unsubscribe.
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ final Set<String> pipeOnlyTopicNames;
+ if (Objects.nonNull(consensusBroker)) {
+ pipeOnlyTopicNames = new java.util.HashSet<>(topicNames);
+ pipeOnlyTopicNames.removeIf(consensusBroker::hasQueue);
+ } else {
+ pipeOnlyTopicNames = topicNames;
+ }
+
+ if (pipeOnlyTopicNames.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
return Collections.emptyList();
}
- return broker.fetchTopicNamesToUnsubscribe(topicNames);
+ return pipeBroker.fetchTopicNamesToUnsubscribe(pipeOnlyTopicNames);
}
/////////////////////////////// broker ///////////////////////////////
public boolean isBrokerExist(final String consumerGroupId) {
- return consumerGroupIdToSubscriptionBroker.containsKey(consumerGroupId);
+ return consumerGroupIdToPipeBroker.containsKey(consumerGroupId)
+ || consumerGroupIdToConsensusBroker.containsKey(consumerGroupId);
}
public void createBrokerIfNotExist(final String consumerGroupId) {
- consumerGroupIdToSubscriptionBroker.computeIfAbsent(consumerGroupId, SubscriptionBroker::new);
- LOGGER.info("Subscription: create broker bound to consumer group [{}]", consumerGroupId);
+ consumerGroupIdToPipeBroker.computeIfAbsent(consumerGroupId, SubscriptionBroker::new);
+ LOGGER.info("Subscription: create pipe broker bound to consumer group [{}]", consumerGroupId);
}
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279143514
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertTabletNode;
+
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/** Converts IoTConsensus WAL log entries (InsertNode) to Tablet format for subscription. */
+public class ConsensusLogToTabletConverter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusLogToTabletConverter.class);
+
+ private final TreePattern treePattern;
+ private final TablePattern tablePattern;
+ private final Pattern tableColumnPattern;
+
+ /**
+ * The actual database name of the DataRegion this converter processes (table-model format without
+ * "root." prefix). Null for tree-model topics.
+ */
+ private final String databaseName;
+
+ public ConsensusLogToTabletConverter(
+ final TreePattern treePattern,
+ final TablePattern tablePattern,
+ final Pattern tableColumnPattern,
+ final String databaseName) {
+ this.treePattern = treePattern;
+ this.tablePattern = tablePattern;
+ this.tableColumnPattern = tableColumnPattern;
+ this.databaseName = databaseName;
+ }
+
+ public String getDatabaseName() {
+ return databaseName;
+ }
+
+ static String safeDeviceIdForLog(final InsertNode node) {
+ try {
+ final Object deviceId = node.getDeviceID();
+ return deviceId != null ? deviceId.toString() : "null";
+ } catch (final Exception e) {
+ return "N/A(" + node.getType() + ")";
+ }
+ }
+
+ public List<Tablet> convert(final InsertNode insertNode) {
+ if (Objects.isNull(insertNode)) {
+ return Collections.emptyList();
+ }
+
+ final PlanNodeType nodeType = insertNode.getType();
+ if (nodeType == null) {
+ LOGGER.warn("InsertNode type is null, skipping conversion");
+ return Collections.emptyList();
+ }
+
+ LOGGER.debug(
+ "ConsensusLogToTabletConverter: converting InsertNode type={}, deviceId={}",
+ nodeType,
+ safeDeviceIdForLog(insertNode));
+
+ switch (nodeType) {
+ case INSERT_ROW:
+ return convertInsertRowNode((InsertRowNode) insertNode);
+ case INSERT_TABLET:
+ return convertInsertTabletNode((InsertTabletNode) insertNode);
+ case INSERT_ROWS:
+ return convertInsertRowsNode((InsertRowsNode) insertNode);
+ case INSERT_ROWS_OF_ONE_DEVICE:
+ return convertInsertRowsOfOneDeviceNode((InsertRowsOfOneDeviceNode) insertNode);
+ case INSERT_MULTI_TABLET:
+ return convertInsertMultiTabletsNode((InsertMultiTabletsNode) insertNode);
+ case RELATIONAL_INSERT_ROW:
+ return convertRelationalInsertRowNode((RelationalInsertRowNode) insertNode);
+ case RELATIONAL_INSERT_TABLET:
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) insertNode);
+ case RELATIONAL_INSERT_ROWS:
+ return convertRelationalInsertRowsNode((RelationalInsertRowsNode) insertNode);
+ default:
+ LOGGER.debug("Unsupported InsertNode type for subscription: {}", nodeType);
+ return Collections.emptyList();
+ }
+ }
+
+ // ======================== Tree Model Conversion ========================
+
+ private List<Tablet> convertInsertRowNode(final InsertRowNode node) {
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final long time = node.getTime();
+
+ // Determine which columns match the pattern
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final Object[] values = node.getValues();
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Build Tablet with matched columns
+ final int columnCount = matchedColumnIndices.size();
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ final Tablet tablet = new Tablet(deviceId.toString(), schemas, 1 /* maxRowNumber */);
+ tablet.addTimestamp(0, time);
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = matchedColumnIndices.get(i);
+ final Object value = values[originalColIdx];
+ if (value == null) {
+ if (tablet.getBitMaps() == null) {
+ tablet.initBitMaps();
+ }
+ tablet.getBitMaps()[i].mark(0);
+ } else {
+ addValueToTablet(tablet, 0, i, dataTypes[originalColIdx], value);
+ }
+ }
+ tablet.setRowSize(1);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
+ if (node instanceof RelationalInsertTabletNode) {
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) node);
+ }
+
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final long[] times = node.getTimes();
+ final Object[] columns = node.getColumns();
+ final BitMap[] bitMaps = node.getBitMaps();
+ final int rowCount = node.getRowCount();
+
+ // Column filtering
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final int columnCount = matchedColumnIndices.size();
+ final boolean allColumnsMatch = (columnCount == measurements.length);
+
+ // Build schemas (always needed)
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ // Build column arrays and bitmaps using bulk copy
+ final long[] newTimes = Arrays.copyOf(times, rowCount);
+ final Object[] newColumns = new Object[columnCount];
+ final BitMap[] newBitMaps = new BitMap[columnCount];
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = allColumnsMatch ? i : matchedColumnIndices.get(i);
+ newColumns[i] = copyColumnArray(dataTypes[originalColIdx], columns[originalColIdx], rowCount);
+ if (bitMaps != null && bitMaps[originalColIdx] != null) {
+ newBitMaps[i] = new BitMap(rowCount);
+ BitMap.copyOfRange(bitMaps[originalColIdx], 0, newBitMaps[i], 0, rowCount);
+ }
+ }
Review Comment:
Yes. The copied arrays detach the converted Tablet from the original InsertNode/WAL buffers and keep only the matched columns. This is consistent with the pipe path. The pipe fast path avoids this by sending the serialized InsertNode directly, but we can't send InsertNode directly for subscription.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279131397
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -200,41 +522,183 @@ public void bindPrefetchingQueue(final SubscriptionSinkSubtask subtask) {
prefetchingQueueCount.invalidate();
}
- public void updateCompletedTopicNames(final String consumerGroupId, final String topicName) {
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
+ public void bindConsensusPrefetchingQueue(
+ final String consumerGroupId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ consumerGroupIdToConsensusBroker
+ .compute(
+ consumerGroupId,
+ (id, broker) -> {
+ if (Objects.isNull(broker)) {
+ LOGGER.info(
+ "Subscription: consensus broker bound to consumer group [{}] does not exist, create new for binding consensus prefetching queue",
+ consumerGroupId);
+ return new ConsensusSubscriptionBroker(consumerGroupId);
+ }
+ return broker;
+ })
+ .bindConsensusPrefetchingQueue(
+ topicName,
+ orderMode,
+ consensusGroupId,
+ serverImpl,
+ retentionPolicy,
+ converter,
+ commitManager,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+ prefetchingQueueCount.invalidate();
+ }
+
+ public void refreshConsensusQueueOrderMode(final String topicName, final String orderMode) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: refreshing consensus queue order-mode for topic [{}] to [{}]",
+ topicName,
+ orderMode);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.refreshConsensusQueueOrderMode(topicName, orderMode);
+ }
+ }
+
+ public void unbindConsensusPrefetchingQueue(
+ final String consumerGroupId, final String topicName) {
+ final ConsensusSubscriptionBroker broker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
if (Objects.isNull(broker)) {
LOGGER.warn(
- "Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
+ "Subscription: consensus broker bound to consumer group [{}] does not exist",
+ consumerGroupId);
return;
}
- broker.updateCompletedTopicNames(topicName);
+ broker.unbindConsensusPrefetchingQueue(topicName);
+ prefetchingQueueCount.invalidate();
+ }
+
+ public void unbindByRegion(final ConsensusGroupId regionId) {
+ int totalClosed = 0;
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ totalClosed += broker.unbindByRegion(regionId);
+ }
+ if (totalClosed > 0) {
+ prefetchingQueueCount.invalidate();
+ LOGGER.info(
+ "Subscription: unbound {} consensus prefetching queue(s) for removed region [{}]",
+ totalClosed,
+ regionId);
+ }
+ }
+
+ /**
+ * Activates or deactivates all consensus prefetching queues bound to {@code regionId} across all
+ * consumer groups. Called on leader migration to ensure only the preferred writer serves
+ * subscription data.
+ */
+ public void setActiveForRegion(final ConsensusGroupId regionId, final boolean active) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: setActiveForRegion regionId={}, active={}", regionId, active);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.setActiveForRegion(regionId, active);
+ }
+ }
+
+ public void setActiveWritersForRegion(
+ final ConsensusGroupId regionId, final Set<Integer> activeWriterNodeIds) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: setActiveWritersForRegion regionId={}, activeWriterNodeIds={}",
+ regionId,
+ activeWriterNodeIds);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.setActiveWritersForRegion(regionId, activeWriterNodeIds);
+ }
+ }
+
+ public void applyRuntimeStateForRegion(
+ final ConsensusGroupId regionId, final ConsensusRegionRuntimeState runtimeState) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: applyRuntimeStateForRegion regionId={}, runtimeState={}",
+ regionId,
+ runtimeState);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.applyRuntimeStateForRegion(regionId, runtimeState);
+ }
+ }
+
+ public void abortConsensusPendingSeeksForRuntimeStop() {
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.abortPendingSeeksForRuntimeStop();
+ }
+ }
+
+ public void updateCompletedTopicNames(final String consumerGroupId, final String topicName) {
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
+ LOGGER.warn(
+ "Subscription: pipe broker bound to consumer group [{}] does not exist", consumerGroupId);
+ return;
+ }
+ pipeBroker.updateCompletedTopicNames(topicName);
}
public void unbindPrefetchingQueue(final String consumerGroupId, final String topicName) {
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ // Try consensus broker first
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ consensusBroker.unbindConsensusPrefetchingQueue(topicName);
+ prefetchingQueueCount.invalidate();
+ return;
+ }
+ // Fall back to pipe broker
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
LOGGER.warn(
"Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
return;
}
- broker.unbindPrefetchingQueue(topicName);
+ pipeBroker.unbindPrefetchingQueue(topicName);
Review Comment:
Extracted the shared broker operations into ISubscriptionBroker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3246263985
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionPullConsumer.java:
##########
@@ -135,6 +141,24 @@ public synchronized void close() {
return;
}
+ // flush all processors and commit any remaining buffered messages
+ if (!processors.isEmpty()) {
+ final List<SubscriptionMessage> flushed = new ArrayList<>();
+ for (final SubscriptionMessageProcessor processor : processors) {
+ final List<SubscriptionMessage> out = processor.flush();
+ if (out != null) {
+ flushed.addAll(out);
+ }
+ }
+ if (!flushed.isEmpty() && autoCommit) {
Review Comment:
Fixed. Manual-commit close no longer silently drops messages flushed from processors. The consumer now drains buffered processor output, preserves user-visible drained messages, and rejects close with a clear exception until the application drains/commits them explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3278693902
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/subscription/SubscriptionInfo.java:
##########
@@ -258,6 +282,228 @@ private void checkBeforeAlteringTopicInternal(TopicMeta topicMeta) throws Subscr
throw new SubscriptionException(exceptionMessage);
}
+ private Map<String, String> safeTopicAttributes(@Nullable final Map<String, String> attributes) {
+ return Objects.nonNull(attributes) ? attributes : Collections.emptyMap();
+ }
+
+ private void validateTopicConfig(final TopicConfig topicConfig) throws SubscriptionException {
+ final String mode = topicConfig.getMode();
+ if (!TopicConfig.isValidMode(mode)) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, unsupported %s=%s, expected one of [%s, %s, %s]",
+ TopicConstant.MODE_KEY,
+ mode,
+ TopicConstant.MODE_SNAPSHOT_VALUE,
+ TopicConstant.MODE_LIVE_VALUE,
+ TopicConstant.MODE_CONSENSUS_VALUE);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ validateConsensusProtocolSupport(topicConfig);
+
+ if (topicConfig.isConsensusMode() && !topicConfig.isRecordFormat()) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, %s=%s only supports %s=%s",
+ TopicConstant.MODE_KEY,
+ TopicConstant.MODE_CONSENSUS_VALUE,
+ TopicConstant.FORMAT_KEY,
+ TopicConstant.FORMAT_RECORD_HANDLER_VALUE);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ final String orderMode = topicConfig.getOrderMode();
+ if (!TopicConfig.isValidOrderMode(orderMode)) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, unsupported %s=%s, expected one of [%s, %s, %s]",
+ TopicConstant.ORDER_MODE_KEY,
+ orderMode,
+ TopicConstant.ORDER_MODE_LEADER_ONLY_VALUE,
+ TopicConstant.ORDER_MODE_MULTI_WRITER_VALUE,
+ TopicConstant.ORDER_MODE_PER_WRITER_VALUE);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ validateConsensusTableColumnPattern(topicConfig);
+ validateConsensusTopicRetentionConfig(topicConfig);
+ }
+
+ private void validateConsensusProtocolSupport(final TopicConfig topicConfig)
+ throws SubscriptionException {
+ if (!topicConfig.isConsensusMode()) {
+ return;
+ }
+
+ final String actualProtocol = String.valueOf(CONF.getDataRegionConsensusProtocolClass());
+ if (ConsensusFactory.IOT_CONSENSUS.equals(actualProtocol)) {
+ return;
+ }
+
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, %s=%s is only supported when %s=%s, but current value is %s",
+ TopicConstant.MODE_KEY,
+ TopicConstant.MODE_CONSENSUS_VALUE,
+ DATA_REGION_CONSENSUS_PROTOCOL_CLASS_KEY,
+ ConsensusFactory.IOT_CONSENSUS,
+ actualProtocol);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ private void validateConsensusTableColumnPattern(final TopicConfig topicConfig)
+ throws SubscriptionException {
+ if (!topicConfig.hasAttribute(TopicConstant.COLUMN_KEY)) {
+ return;
+ }
+
+ if (!topicConfig.isTableTopic()) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, %s is only supported for table topics",
+ TopicConstant.COLUMN_KEY);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ if (!isConsensusBasedTopicConfig(topicConfig)) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, %s is only supported for consensus table topics",
+ TopicConstant.COLUMN_KEY);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ final String columnPattern =
+ topicConfig.getStringOrDefault(
+ TopicConstant.COLUMN_KEY, TopicConstant.COLUMN_DEFAULT_VALUE);
+ try {
+ Pattern.compile(columnPattern);
+ } catch (final PatternSyntaxException e) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, illegal %s=%s, detail: %s",
+ TopicConstant.COLUMN_KEY, columnPattern, e.getMessage());
+ LOGGER.warn(exceptionMessage, e);
+ throw new SubscriptionException(exceptionMessage);
+ }
+ }
+
+ private boolean isConsensusBasedTopicConfig(final TopicConfig topicConfig) {
+ return topicConfig.isConsensusMode();
+ }
+
+ private void validateConsensusTopicRetentionConfig(final TopicConfig topicConfig)
+ throws SubscriptionException {
+ if (!topicConfig.hasAttribute(TopicConstant.RETENTION_BYTES_KEY)
+ && !topicConfig.hasAttribute(TopicConstant.RETENTION_MS_KEY)) {
+ return;
+ }
+
+ if (!isConsensusBasedTopicConfig(topicConfig)) {
+ final String exceptionMessage =
+ String.format(
+ "Failed to create or alter topic, %s and %s are only supported for consensus topics",
+ TopicConstant.RETENTION_BYTES_KEY, TopicConstant.RETENTION_MS_KEY);
+ LOGGER.warn(exceptionMessage);
+ throw new SubscriptionException(exceptionMessage);
+ }
+
+ validateRetentionValue(topicConfig, TopicConstant.RETENTION_BYTES_KEY);
+ validateRetentionValue(topicConfig, TopicConstant.RETENTION_MS_KEY);
Review Comment:
Kafka uses raw numeric values for retention.bytes and retention.ms, the unit is encoded in the config key. The current behavior in our code is the same, so I kept it unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279180407
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = inFlightEvents.get(new Pair<>(consumerId, commitContext));
+ if (Objects.isNull(event)) {
+ if (isCommitContextOutdated(commitContext)) {
+ return generateOutdatedErrorResponse();
+ }
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
+ this, consumerId, commitContext));
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ // ======================== Prefetch Round Drive ========================
+
+ private static final long WAL_GAP_RETRY_SLEEP_MS = 10L;
+ private static final long WAL_GAP_WAIT_LOG_INTERVAL_MS = 5_000L;
+
+ private static final long PREFETCH_STATS_LOG_INTERVAL_MS = 5_000L;
+
+ public PrefetchRoundResult drivePrefetchOnce() {
+ if (applyPendingSeekRequestIfNecessary()) {
+ return closeRequested ? PrefetchRoundResult.dormant() : PrefetchRoundResult.rescheduleNow();
+ }
+
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !prefetchInitialized) {
+ return PrefetchRoundResult.dormant();
+ }
+
+ logPeriodicStatsIfNecessary();
+
+ final long currentSeekGeneration = seekGeneration.get();
+ if (currentSeekGeneration != observedSeekGeneration) {
+ resetRoundStateForSeek(currentSeekGeneration);
+ }
+
+ applyPendingSubscriptionWalReset(observedSeekGeneration);
+ recycleInFlightEvents();
+
+ if (!isActive || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return computeIdleRoundResult();
+ }
+
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+ final int batchMaxDelayMs = config.getSubscriptionConsensusBatchMaxDelayInMs();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+
+ final List<IndexedConsensusRequest> batch = drainPendingEntries(maxWalEntries);
+ if (!batch.isEmpty()) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: drained {} entries from pendingEntries, "
+ + "first searchIndex={}, last searchIndex={}, nextExpected={}, "
+ + "prefetchingQueueSize={}",
+ this,
+ batch.size(),
+ batch.get(0).getSearchIndex(),
+ batch.get(batch.size() - 1).getSearchIndex(),
+ nextExpectedSearchIndex.get(),
+ prefetchingQueue.size());
+
+ final boolean batchAccepted =
+ accumulateFromPending(
+ batch, lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes);
+ if (!batchAccepted) {
+ if (pendingWalGapRetryRequested) {
+ // Once a drained batch hits an unresolved WAL gap, the affected suffix falls back to
+ // the WAL path on later rounds instead of being requeued into the bounded pending path.
+ return PrefetchRoundResult.rescheduleAfter(WAL_GAP_RETRY_SLEEP_MS);
+ }
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ }
+
+ if (batch.isEmpty() && lingerBatch.isEmpty()) {
+ tryCatchUpFromWAL(observedSeekGeneration);
+ }
+
+ if (!drainBufferedRealtimeLanes(
+ lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes)) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerElapsedMs = System.currentTimeMillis() - lingerBatch.firstTabletTimeMs;
+ if (lingerElapsedMs >= batchMaxDelayMs) {
+ if (seekGeneration.get() != observedSeekGeneration) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: time-based flush, {} tablets lingered for {}ms "
+ + "(threshold={}ms)",
+ this,
+ lingerBatch.tablets.size(),
+ lingerElapsedMs,
+ batchMaxDelayMs);
+ flushBatch(lingerBatch, observedSeekGeneration);
+ }
+ }
+
+ maybeInjectWatermark();
+ return computeIdleRoundResult();
+ } catch (final Throwable fatal) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: prefetch round failed " + "(type={}, message={})",
+ this,
+ fatal.getClass().getName(),
+ fatal.getMessage(),
+ fatal);
+ if (fatal instanceof VirtualMachineError) {
+ markClosed();
+ return PrefetchRoundResult.dormant();
+ }
+ return PrefetchRoundResult.rescheduleAfter(100L);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private void logPeriodicStatsIfNecessary() {
+ final long nowMs = System.currentTimeMillis();
+ if (nowMs - lastStatsLogTimeMs < PREFETCH_STATS_LOG_INTERVAL_MS) {
+ return;
+ }
+
+ final long currentPendingAcceptedEntries = pendingPathAcceptedEntries.get();
+ final long currentWalAcceptedEntries = walPathAcceptedEntries.get();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: periodic stats, lag={}, pendingDelta={}, walDelta={}, "
+ + "pendingTotal={}, walTotal={}, pendingQueueSize={}, prefetchingQueueSize={}, "
+ + "inFlightEventsSize={}, realtimeLaneCount={}, walHasNext={}, isActive={}, subtaskScheduled={}",
+ this,
+ getLag(),
+ currentPendingAcceptedEntries - lastPendingAcceptedEntries,
+ currentWalAcceptedEntries - lastWalAcceptedEntries,
+ currentPendingAcceptedEntries,
+ currentWalAcceptedEntries,
+ pendingEntries.size(),
+ prefetchingQueue.size(),
+ inFlightEvents.size(),
+ realtimeEntriesByLane.size(),
+ hasReadableWalEntries(),
+ isActive,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ lastStatsLogTimeMs = nowMs;
+ lastPendingAcceptedEntries = currentPendingAcceptedEntries;
+ lastWalAcceptedEntries = currentWalAcceptedEntries;
+ }
+
+ private void resetRoundStateForSeek(final long newSeekGeneration) {
+ restorePendingSubscriptionWalCursor(newSeekGeneration);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = newSeekGeneration;
+ }
+
+ private List<IndexedConsensusRequest> drainPendingEntries(final int maxWalEntries) {
+ final List<IndexedConsensusRequest> batch = new ArrayList<>();
+ IndexedConsensusRequest next;
+ while (batch.size() < maxWalEntries && (next = pendingEntries.poll()) != null) {
+ batch.add(next);
+ }
+ return batch;
+ }
+
+ private PrefetchRoundResult computeIdleRoundResult() {
+ if (isClosed || !prefetchInitialized || !isActive) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (hasImmediatePrefetchableWork()) {
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ long delayMs = Long.MAX_VALUE;
+ if (hasHistoricalWalLag()) {
+ delayMs = Math.min(delayMs, WAL_GAP_RETRY_SLEEP_MS);
+ }
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerDelayMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusBatchMaxDelayInMs()
+ - (System.currentTimeMillis() - lingerBatch.firstTabletTimeMs);
+ delayMs = Math.min(delayMs, Math.max(1L, lingerDelayMs));
+ }
+
+ final long watermarkDelayMs = computeWatermarkDelayMs();
+ if (watermarkDelayMs > 0L) {
+ delayMs = Math.min(delayMs, watermarkDelayMs);
+ }
+
+ if (!inFlightEvents.isEmpty()) {
+ delayMs =
+ Math.min(
+ delayMs,
+ SubscriptionConfig.getInstance().getSubscriptionRecycleUncommittedEventIntervalMs());
+ }
+
+ return delayMs == Long.MAX_VALUE
+ ? PrefetchRoundResult.dormant()
+ : PrefetchRoundResult.rescheduleAfter(delayMs);
+ }
+
+ private long computeWatermarkDelayMs() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return -1L;
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0L) {
+ return -1L;
+ }
+ if (lastWatermarkEmitTimeMs == 0L) {
+ return 1L;
+ }
+ final long elapsedMs = System.currentTimeMillis() - lastWatermarkEmitTimeMs;
+ return elapsedMs >= intervalMs ? 1L : Math.max(1L, intervalMs - elapsedMs);
+ }
+
+ private boolean hasImmediatePrefetchableWork() {
+ return !pendingEntries.isEmpty() || !realtimeEntriesByLane.isEmpty() || hasReadableWalEntries();
+ }
+
+ private boolean hasHistoricalWalLag() {
+ return nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex();
+ }
+
+ /**
+ * Accumulates tablets from pending entries into the linger buffer. When pending replay outruns
+ * the local WAL reader, this method backfills the local-index gap from WAL before continuing.
+ *
+ * @return false if the batch became stale because seek generation changed while flushing
+ */
+ private static boolean hasLocalSearchIndex(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() >= 0;
+ }
+
+ private boolean isBeforeLocalCursor(final IndexedConsensusRequest request) {
+ return hasLocalSearchIndex(request) && request.getSearchIndex() < nextExpectedSearchIndex.get();
+ }
+
+ private void advanceLocalCursorIfPresent(final IndexedConsensusRequest request) {
+ if (hasLocalSearchIndex(request)) {
+ nextExpectedSearchIndex.set(request.getSearchIndex() + 1);
+ }
+ }
+
+ private boolean appendRealtimeRequest(
+ final IndexedConsensusRequest request,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean fromPending) {
+ final PreparedEntry preparedEntry = prepareEntry(request);
+ if (Objects.isNull(preparedEntry)) {
+ return true;
+ }
+ if (!appendPreparedEntryViaRealtimeLane(
+ batchState, preparedEntry, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+ if (fromPending) {
+ markAcceptedFromPending();
+ } else {
+ markAcceptedFromWal();
+ }
+ return true;
+ }
+
+ private boolean accumulateFromPending(
+ final List<IndexedConsensusRequest> batch,
+ final DeliveryBatchState lingerBatch,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+
+ int processedCount = 0;
+ int skippedCount = 0;
+
+ for (int index = 0; index < batch.size(); index++) {
+ final IndexedConsensusRequest request = batch.get(index);
+ final long searchIndex = request.getSearchIndex();
+
+ // Only local-indexed requests participate in the internal WAL read cursor.
+ final long expected = nextExpectedSearchIndex.get();
+ if (hasLocalSearchIndex(request) && searchIndex > expected) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: gap detected, expected={}, got={}. "
+ + "Filling {} entries from WAL.",
+ this,
+ expected,
+ searchIndex,
+ searchIndex - expected);
+ if (!fillGapFromWAL(
+ expected,
+ searchIndex,
+ lingerBatch,
+ expectedSeekGeneration,
+ maxTablets,
+ maxBatchBytes)) {
+ return false;
+ }
+ }
+
+ if (isBeforeLocalCursor(request)) {
+ skippedCount++;
+ continue;
+ }
+
+ if (shouldSkipForRecoveryProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ request, lingerBatch, expectedSeekGeneration, maxTablets, maxBatchBytes, true)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(request);
+ processedCount++;
+ advanceLocalCursorIfPresent(request);
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: accumulate complete, batchSize={}, processed={}, "
+ + "skipped={}, lingerTablets={}, nextExpected={}",
+ this,
+ batch.size(),
+ processedCount,
+ skippedCount,
+ lingerBatch.tablets.size(),
+ nextExpectedSearchIndex.get());
+
+ return true;
+ }
+
+ /**
+ * Fills a gap in the pending queue by reading entries from WAL so the internal local replay
+ * cursor stays contiguous even when pending delivery jumps ahead of the WAL iterator.
+ *
+ * <p>Temporary WAL visibility lag is treated as a normal back-pressure condition: once a drained
+ * pending batch encounters an unresolved local-index gap, the queue backs off and lets the
+ * affected suffix fall back to the WAL path on later rounds. This keeps replay contiguous without
+ * requeueing the drained batch back into the bounded pending queue.
+ *
+ * @return false if gap fill had to stop because the current batch became stale or the queue was
+ * interrupted/closed
+ */
+ private boolean fillGapFromWAL(
+ final long fromIndex,
+ final long toIndex,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ pendingWalGapRetryRequested = false;
+ resetSubscriptionWALPosition(fromIndex);
+ if (seekGeneration.get() != expectedSeekGeneration || isClosed) {
+ return false;
+ }
+ if (!pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, Integer.MAX_VALUE, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final long nextExpected = nextExpectedSearchIndex.get();
+ if (nextExpected >= toIndex) {
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ return true;
+ }
+
+ final long nowMs = System.currentTimeMillis();
+ if (walGapWaitStartTimeMs == 0L) {
+ walGapWaitStartTimeMs = nowMs;
+ }
+ if (lastWalGapWaitLogTimeMs == 0L
+ || nowMs - lastWalGapWaitLogTimeMs >= WAL_GAP_WAIT_LOG_INTERVAL_MS) {
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: waiting {}ms for WAL gap [{}, {}) to become visible, "
+ + "currentNextExpected={}, currentWalIndex={}, seekGeneration={}",
+ this,
+ nowMs - walGapWaitStartTimeMs,
+ nextExpected,
+ toIndex,
+ nextExpected,
+ consensusReqReader.getCurrentSearchIndex(),
+ expectedSeekGeneration);
+ lastWalGapWaitLogTimeMs = nowMs;
+ }
+ onWalGapRetryScheduled();
+ pendingWalGapRetryRequested = true;
+ return false;
+ }
+
+ /**
+ * Try catch-up from WAL when the pending queue was empty. This handles cold-start or scenarios
+ * where the subscription started after data was already written.
+ */
+ private void tryCatchUpFromWAL(final long expectedSeekGeneration) {
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+
+ final DeliveryBatchState batchState = new DeliveryBatchState();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ final boolean accepted =
+ pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, maxWalEntries, maxTablets, maxBatchBytes);
+ if (!accepted) {
+ return;
+ }
+
+ if (!batchState.isEmpty()) {
+ flushBatch(batchState, expectedSeekGeneration);
+ }
+ }
+
+ private boolean pumpFromSubscriptionWAL(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxWalEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return true;
+ }
+
+ subscriptionWALIterator.refresh();
+ ensureSubscriptionWalReadable();
+
+ int entriesRead = 0;
+ while (entriesRead < maxWalEntries
+ && subscriptionWALIterator.hasNext()
+ && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ try {
+ final IndexedConsensusRequest walEntry = subscriptionWALIterator.next();
+ entriesRead++;
+
+ if (isBeforeLocalCursor(walEntry)) {
+ continue;
+ }
+ if (shouldSkipForRecoveryProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ walEntry, batchState, expectedSeekGeneration, maxTablets, maxBatchBytes, false)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(walEntry);
+ advanceLocalCursorIfPresent(walEntry);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error reading subscription WAL", this, e);
+ break;
+ }
+ }
+
+ if (entriesRead > 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL read {} entries, nextExpectedSearchIndex={}",
+ this,
+ entriesRead,
+ nextExpectedSearchIndex.get());
+ }
+ return true;
+ }
+
+ private void ensureSubscriptionWalReadable() {
+ if (Objects.isNull(subscriptionWALIterator)
+ || subscriptionWALIterator.hasNext()
+ || !(consensusReqReader instanceof WALNode)) {
+ return;
+ }
+
+ final long currentWalIndex = consensusReqReader.getCurrentSearchIndex();
+ if (nextExpectedSearchIndex.get() > currentWalIndex) {
+ return;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL exhausted at {} while current WAL is {}. "
+ + "Rolling WAL file to expose current-file entries.",
+ this,
+ nextExpectedSearchIndex.get(),
+ currentWalIndex);
+ ((WALNode) consensusReqReader).rollWALFile();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ if (Objects.nonNull(subscriptionWALIterator)) {
+ subscriptionWALIterator.refresh();
+ }
+ }
+
+ private void resetSubscriptionWALPosition(final long startSearchIndex) {
+ closeSubscriptionWALIterator();
+ subscriptionWALIterator = createSubscriptionWALIterator(startSearchIndex);
+ }
+
+ protected ProgressWALIterator createSubscriptionWALIterator(final long startSearchIndex) {
+ if (consensusReqReader instanceof WALNode) {
+ return new ProgressWALIterator((WALNode) consensusReqReader, startSearchIndex);
+ }
+ return null;
+ }
+
+ protected void onWalGapRetryScheduled() {}
+
+ private boolean hasReadableWalEntries() {
+ return Objects.nonNull(subscriptionWALIterator) && subscriptionWALIterator.hasNext();
+ }
+
+ private void requestSubscriptionWalReset(
+ final long targetSearchIndex, final long seekGenerationValue) {
+ pendingSubscriptionWalResetSearchIndex = targetSearchIndex;
+ pendingSubscriptionWalResetGeneration = seekGenerationValue;
+ }
+
+ private void applyPendingSubscriptionWalReset(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ resetSubscriptionWALPosition(pendingSubscriptionWalResetSearchIndex);
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ }
+
+ private void restorePendingSubscriptionWalCursor(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ // A seek can land in the middle of a prefetch iteration. Restore the local cursor to the
+ // pending seek target before resuming under the new generation so stale in-flight work does
+ // not permanently advance the historical replay frontier.
+ nextExpectedSearchIndex.set(pendingSubscriptionWalResetSearchIndex);
+ }
+
+ private void closeSubscriptionWALIterator() {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return;
+ }
+ try {
+ subscriptionWALIterator.close();
+ } catch (final IOException e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error closing subscription WAL iterator", this, e);
+ } finally {
+ subscriptionWALIterator = null;
+ }
+ }
+
+ /**
+ * Deserializes the IConsensusRequest entries within an IndexedConsensusRequest to produce an
+ * InsertNode. WAL entries are typically stored as IoTConsensusRequest (serialized ByteBuffers),
+ * and a single logical write may be split across multiple fragments (SearchNode). This method
+ * handles both cases.
+ *
+ * <p>The deserialization follows the same pattern as {@code
+ * DataRegionStateMachine.grabPlanNode()}.
+ */
+ private InsertNode deserializeToInsertNode(final IndexedConsensusRequest indexedRequest) {
+ final List<SearchNode> searchNodes = new ArrayList<>();
+ PlanNode nonSearchNode = null;
+
+ for (final IConsensusRequest req : indexedRequest.getRequests()) {
+ PlanNode planNode;
+ try {
+ if (req instanceof IoTConsensusRequest) {
+ // WAL entries read from file are wrapped as IoTConsensusRequest (ByteBuffer)
+ planNode = WALEntry.deserializeForConsensus(req.serializeToByteBuffer());
+ } else if (req instanceof InsertNode) {
+ // In-memory entries (not yet flushed to WAL file) may already be PlanNode
+ planNode = (PlanNode) req;
+ } else {
+ // ByteBufferConsensusRequest or unknown
+ planNode = PlanNodeType.deserialize(req.serializeToByteBuffer());
+ }
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to deserialize IConsensusRequest "
+ + "(type={}) in searchIndex={}: {}",
+ this,
+ req.getClass().getSimpleName(),
+ indexedRequest.getSearchIndex(),
+ e.getMessage(),
+ e);
+ continue;
+ }
+
+ if (planNode instanceof SearchNode) {
+ final SearchNode searchNode = (SearchNode) planNode;
+ searchNode.setSearchIndex(indexedRequest.getSearchIndex());
+ if (indexedRequest.getSyncIndex() >= 0) {
+ searchNode.setSyncIndex(indexedRequest.getSyncIndex());
+ }
+ if (indexedRequest.getPhysicalTime() > 0) {
+ searchNode.setPhysicalTime(indexedRequest.getPhysicalTime());
+ }
+ if (indexedRequest.getNodeId() >= 0) {
+ searchNode.setNodeId(indexedRequest.getNodeId());
+ }
+ searchNodes.add(searchNode);
+ } else {
+ nonSearchNode = planNode;
+ }
+ }
+
+ // Merge split SearchNode fragments (same pattern as DataRegionStateMachine.grabPlanNode)
+ if (!searchNodes.isEmpty()) {
+ final PlanNode merged = searchNodes.get(0).merge(searchNodes);
+ if (merged instanceof InsertNode) {
+ final InsertNode mergedInsert = (InsertNode) merged;
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: deserialized merged InsertNode for searchIndex={}, "
+ + "type={}, deviceId={}, searchNodeCount={}",
+ this,
+ indexedRequest.getSearchIndex(),
+ mergedInsert.getType(),
+ ConsensusLogToTabletConverter.safeDeviceIdForLog(mergedInsert),
+ searchNodes.size());
+
+ return mergedInsert;
+ }
+ }
+
+ if (nonSearchNode != null) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: searchIndex={} contains non-InsertNode PlanNode: {}",
+ this,
+ indexedRequest.getSearchIndex(),
+ nonSearchNode.getClass().getSimpleName());
+ }
+
+ return null;
+ }
+
+ private PreparedEntry prepareEntry(final IndexedConsensusRequest indexedRequest) {
+ final InsertNode insertNode = deserializeToInsertNode(indexedRequest);
+ if (Objects.isNull(insertNode)) {
+ return null;
+ }
+
+ final long localSeq =
+ indexedRequest.getProgressLocalSeq() >= 0
+ ? indexedRequest.getProgressLocalSeq()
+ : indexedRequest.getSearchIndex();
+ final long searchIndex = indexedRequest.getSearchIndex();
+ final long physicalTime =
+ indexedRequest.getPhysicalTime() > 0
+ ? indexedRequest.getPhysicalTime()
+ : insertNode.getPhysicalTime();
+ final int writerNodeId =
+ indexedRequest.getNodeId() >= 0 ? indexedRequest.getNodeId() : insertNode.getNodeId();
+
+ trackWriterLane(writerNodeId);
+ final long maxTs = extractMaxTime(insertNode);
+ if (maxTs > maxObservedTimestamp) {
+ maxObservedTimestamp = maxTs;
+ }
+ final List<Tablet> tablets = converter.convert(insertNode);
+ if (tablets.isEmpty()) {
+ return null;
+ }
+
+ return new PreparedEntry(tablets, physicalTime, writerNodeId, localSeq, searchIndex);
+ }
+
+ private static long estimateTabletSize(final Tablet tablet) {
+ return PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet);
+ }
+
+ private void createAndEnqueueEvent(
+ final List<Tablet> tablets, final long startSearchIndex, final long endSearchIndex) {
+ createAndEnqueueEvent(
+ tablets, startSearchIndex, endSearchIndex, endSearchIndex, seekGeneration.get());
+ }
+
+ private boolean createAndEnqueueEvent(
+ final List<Tablet> tablets,
+ final long startSearchIndex,
+ final long endSearchIndex,
+ final long commitLocalSeq,
+ final long expectedSeekGeneration) {
+ if (tablets.isEmpty()) {
+ return true;
+ }
+
+ if (seekGeneration.get() != expectedSeekGeneration) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: skip stale event with searchIndex range [{}, {}], "
+ + "expectedSeekGeneration={}, currentSeekGeneration={}",
+ this,
+ startSearchIndex,
+ endSearchIndex,
+ expectedSeekGeneration,
+ seekGeneration.get());
+ return false;
+ }
+
+ final SubscriptionCommitContext commitContext = buildWriterCommitContext(commitLocalSeq);
+ final WriterId writerId = commitContext.getWriterId();
+ final WriterProgress writerProgress = commitContext.getWriterProgress();
+ commitManager.recordMapping(brokerId, topicName, consensusGroupId, writerId, writerProgress);
+
+ // nextOffset <= 0 means all tablets delivered in single batch
+ // -tablets.size() indicates total count
+ // Use Map<String, List<Tablet>> constructor with actual database name for table model;
+ final TabletsPayload payload =
+ new TabletsPayload(
+ Collections.singletonMap(converter.getDatabaseName(), tablets), -tablets.size());
+
+ final SubscriptionEvent event =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.TABLETS.getType(), payload, commitContext);
+
+ prefetchingQueue.add(event);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: ENQUEUED event with {} tablets, "
+ + "searchIndex range [{}, {}], prefetchQueueSize={}",
+ this,
+ tablets.size(),
+ startSearchIndex,
+ endSearchIndex,
+ prefetchingQueue.size());
+
+ // After enqueuing the data event, control metadata is handled separately from user data.
+ return true;
+ }
+
+ private SubscriptionCommitContext buildWriterCommitContext(final long localSeq) {
+ final int effectiveNodeId =
+ batchWriterNodeId >= 0
+ ? batchWriterNodeId
+ : IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), effectiveNodeId);
+ final WriterProgress writerProgress = new WriterProgress(batchPhysicalTime, localSeq);
+ return new SubscriptionCommitContext(
+ IoTDBDescriptor.getInstance().getConfig().getDataNodeId(),
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ seekGeneration.get(),
+ writerId,
+ writerProgress);
+ }
+
+ private void updateBatchWriterProgress(final long physicalTime, final int writerNodeId) {
+ if (physicalTime > 0) {
+ this.batchPhysicalTime = physicalTime;
+ }
+ if (writerNodeId >= 0) {
+ this.batchWriterNodeId = writerNodeId;
+ }
+ }
+
+ private void resetBatchWriterProgress() {
+ this.batchPhysicalTime = 0L;
+ this.batchWriterNodeId = -1;
+ }
+
+ private long estimateTabletsBytes(final List<Tablet> tablets) {
+ long estimatedBytes = 0L;
+ for (final Tablet tablet : tablets) {
+ estimatedBytes += estimateTabletSize(tablet);
+ }
+ return estimatedBytes;
+ }
+
+ private boolean appendPreparedEntryViaRealtimeLane(
+ final DeliveryBatchState batchState,
+ final PreparedEntry preparedEntry,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ bufferRealtimeEntry(preparedEntry);
+ return drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes);
+ }
+
+ private int getRealtimeBufferedEntryCount() {
+ int count = 0;
+ for (final NavigableMap<Long, PreparedEntry> laneEntries : realtimeEntriesByLane.values()) {
+ count += laneEntries.size();
+ }
+ return count;
+ }
+
+ private boolean drainBufferedRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ while (!realtimeEntriesByLane.isEmpty()) {
+ final int bufferedBefore = getRealtimeBufferedEntryCount();
+ if (!drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final int bufferedAfter = getRealtimeBufferedEntryCount();
+ if (bufferedAfter == 0 || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return true;
+ }
+
+ if (batchState.isEmpty()) {
+ return true;
+ }
+
+ if (!flushBatch(batchState, expectedSeekGeneration)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private boolean canAppendLaneEntry(
+ final DeliveryBatchState batchState,
+ final LaneBufferedEntry entry,
+ final long entryEstimatedBytes,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ final boolean wouldExceedEntryLimit =
+ maxEntries != Integer.MAX_VALUE && batchState.entryCount >= maxEntries;
+ final boolean wouldExceedTabletLimit =
+ !batchState.isEmpty() && batchState.tablets.size() + entry.getTablets().size() > maxTablets;
+ final boolean wouldExceedByteLimit =
+ !batchState.isEmpty() && batchState.estimatedBytes + entryEstimatedBytes > maxBatchBytes;
+ // Keep all consensus subscription modes on a single-writer commit/delivery shape so
+ // SubscriptionCommitContext and RegionProgress remain per-writer.
+ final boolean writerChanged =
+ !batchState.isEmpty() && batchState.writerNodeId != entry.getWriterNodeId();
+ return !(wouldExceedEntryLimit
+ || wouldExceedTabletLimit
+ || wouldExceedByteLimit
+ || writerChanged);
+ }
+
+ private boolean drainRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ return drainLaneEntries(
+ batchState,
+ this::buildRealtimeLaneFrontiers,
+ this::peekRealtimeEntry,
+ entry -> true,
+ (laneId, entry) -> removeRealtimeEntry(laneId, entry.localSeq),
+ Integer.MAX_VALUE,
+ maxTablets,
+ maxBatchBytes,
+ true);
+ }
+
+ private <T extends LaneBufferedEntry> boolean drainLaneEntries(
+ final DeliveryBatchState batchState,
+ final Supplier<PriorityQueue<LaneFrontier>> frontierSupplier,
+ final Function<WriterLaneId, T> headSupplier,
+ final Predicate<T> releasePredicate,
+ final BiConsumer<WriterLaneId, T> removeHeadAction,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean trackLingerTime) {
+ while (true) {
+ final PriorityQueue<LaneFrontier> frontiers = frontierSupplier.get();
+ if (frontiers.isEmpty()) {
+ return true;
+ }
+ final LaneFrontier frontier = frontiers.peek();
+ if (Objects.isNull(frontier) || frontier.isBarrier) {
+ return true;
+ }
+ final T laneHead = headSupplier.apply(frontier.laneId);
+ if (Objects.isNull(laneHead)) {
+ return true;
+ }
+ if (!releasePredicate.test(laneHead)) {
+ return true;
+ }
+
+ final long entryEstimatedBytes = estimateTabletsBytes(laneHead.getTablets());
+ if (!canAppendLaneEntry(
+ batchState, laneHead, entryEstimatedBytes, maxEntries, maxTablets, maxBatchBytes)) {
+ return true;
+ }
+
+ removeHeadAction.accept(frontier.laneId, laneHead);
+ batchState.append(laneHead, entryEstimatedBytes, trackLingerTime);
+ }
+ }
+
+ private boolean flushBatch(
+ final DeliveryBatchState batchState, final long expectedSeekGeneration) {
+ updateBatchWriterProgress(batchState.physicalTime, batchState.writerNodeId);
+ if (!createAndEnqueueEvent(
+ new ArrayList<>(batchState.tablets),
+ batchState.startSearchIndex,
+ batchState.endSearchIndex,
+ batchState.lastLocalSeq,
+ expectedSeekGeneration)) {
+ return false;
+ }
+ resetBatchWriterProgress();
+ batchState.reset();
+ return true;
+ }
+
+ // ======================== Commit (Ack/Nack) ========================
+
+ private boolean canAcceptCommitContext(
+ final SubscriptionCommitContext commitContext, final String action, final boolean silent) {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+ if (Objects.isNull(commitContext) || !commitContext.hasWriterProgress()) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ }
+ return false;
+ }
+ if (!isActive) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ }
+ return false;
+ }
+ return true;
+ }
+
+ public boolean ack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "ack", false)
+ && ackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean ackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ if (!acked.get()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for ack",
+ this,
+ commitContext);
+ }
+ return null;
+ }
+
+ if (ev.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: event {} already committed", this, commitContext);
+ ev.cleanUp(false);
+ return null;
+ }
+
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to advance commit frontier for {}",
+ this,
+ commitContext);
+ return ev;
+ }
+
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+
+ return acked.get();
+ }
+
+ public boolean nack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "nack", false)
+ && nackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ /**
+ * Silent version of ack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean ackSilent(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "ack", true)) {
+ return false;
+ }
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ return ev;
+ }
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+ return acked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private WriterId extractCommitWriterId(final SubscriptionCommitContext commitContext) {
+ final WriterId writerId = commitContext.getWriterId();
+ return Objects.nonNull(writerId) ? writerId : new WriterId(consensusGroupId.toString(), -1);
+ }
+
+ private WriterProgress extractCommitWriterProgress(
+ final SubscriptionCommitContext commitContext) {
+ return commitContext.getWriterProgress();
+ }
+
+ /**
+ * Silent version of nack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean nackSilent(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "nack", true)) {
+ return false;
+ }
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+ return nacked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean nackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for nack",
+ this,
+ commitContext);
+ return null;
+ }
+
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+
+ return nacked.get();
+ }
+
+ // ======================== Recycle ========================
+
+ /** Recycles in-flight events that are pollable (timed out) back to the prefetching queue. */
+ private void recycleInFlightEvents() {
+ for (final Pair<String, SubscriptionCommitContext> key :
+ new ArrayList<>(inFlightEvents.keySet())) {
+ inFlightEvents.compute(
+ key,
+ (k, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ if (ev.pollable()) {
+ ev.nack();
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected during recycle "
+ + "(nackCount={}), force-acking event {}",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: recycled timed-out event {} back to prefetching queue",
+ this,
+ ev);
+ return null;
+ }
+ return ev;
+ });
+ }
+ }
+
+ // ======================== Cleanup ========================
+
+ public void cleanUp() {
+ acquireWriteLock();
+ try {
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ pendingEntries.clear();
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ closeSubscriptionWALIterator();
+
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ // ======================== Seek ========================
+
+ /**
+ * Seeks to the earliest available WAL position. The actual position depends on WAL retention: if
+ * old files have been reclaimed, the earliest available position may be later than 0.
+ */
+ public void seekToBeginning() {
+ seekToResolvedPosition(0L, new RegionProgress(Collections.emptyMap()), "beginning");
+ }
+
+ /**
+ * Seeks to the current WAL write position. After this, only newly written data will be consumed.
+ */
+ public void seekToEnd() {
+ seekToResolvedPosition(
+ consensusReqReader.getCurrentSearchIndex(), computeTailRegionProgress(), "end");
+ }
+
+ public void seekToRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress not supported (no WAL directory)",
+ this);
+ seekToBeginning();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, false);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgress");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekToRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ public void seekAfterRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress not supported (no WAL directory)",
+ this);
+ seekToEnd();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgressAfter");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekAfterRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private synchronized void seekToResolvedPosition(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason) {
+ final PendingSeekRequest request;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ // Fence old commit contexts immediately. The grouped reset itself is applied later by the
+ // prefetch worker so WAL state and queue state still move under the queue's serial context.
+ final boolean previousPrefetchInitialized = prefetchInitialized;
+ final long previousSeekGeneration = seekGeneration.get();
+ final long targetSeekGeneration = seekGeneration.incrementAndGet();
+ request =
+ new PendingSeekRequest(
+ targetSearchIndex,
+ committedRegionProgress,
+ seekReason,
+ previousPrefetchInitialized,
+ previousSeekGeneration,
+ targetSeekGeneration);
+ pendingSeekRequest = request;
+ prefetchInitialized = true;
+ } finally {
+ releaseWriteLock();
+ }
+
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.isNull(subtask)) {
+ failPendingSeekBeforeScheduling(request);
+ request.awaitCompletion();
+ return;
+ }
+
+ subtask.requestWakeupNow();
+ request.awaitCompletion();
+ }
+
+ private boolean applyPendingSeekRequestIfNecessary() {
+ final PendingSeekRequest request = pendingSeekRequest;
+ if (Objects.isNull(request)) {
+ return false;
+ }
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return pendingSeekRequest != null;
+ }
+ pendingSeekRequest = null;
+ if (isClosed || closeRequested) {
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s is closing while applying seek", this)));
+ return true;
+ }
+ applySeekResetUnderWriteLock(request);
+ request.complete();
+ return true;
+ } catch (final RuntimeException e) {
+ request.fail(e);
+ throw e;
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ public void abortPendingSeekForRuntimeStop() {
+ final PendingSeekRequest requestToFail;
+
+ acquireWriteLock();
+ try {
+ requestToFail = pendingSeekRequest;
+ if (Objects.isNull(requestToFail)) {
+ return;
+ }
+ pendingSeekRequest = null;
+ prefetchInitialized = requestToFail.previousPrefetchInitialized;
+ if (seekGeneration.get() == requestToFail.targetSeekGeneration) {
+ seekGeneration.set(requestToFail.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: aborted pending seek({}) during runtime stop, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ requestToFail.seekReason,
+ true,
+ requestToFail.previousPrefetchInitialized,
+ requestToFail.targetSeekGeneration,
+ requestToFail.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ requestToFail.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s runtime stopped before seek(%s) was applied",
+ this, requestToFail.seekReason)));
+ }
+
+ private void failPendingSeekBeforeScheduling(final PendingSeekRequest request) {
+ final boolean closing;
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return;
+ }
+ closing = isClosed || closeRequested;
+ pendingSeekRequest = null;
+ prefetchInitialized = request.previousPrefetchInitialized;
+ if (seekGeneration.get() == request.targetSeekGeneration) {
+ seekGeneration.set(request.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: failed to schedule seek({}) because {}, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ request.seekReason,
+ closing ? "the queue is closing" : "prefetch runtime is unavailable",
+ true,
+ request.previousPrefetchInitialized,
+ request.targetSeekGeneration,
+ request.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ closing
+ ? "ConsensusPrefetchingQueue %s is closing before seek(%s) can be scheduled"
+ : "ConsensusPrefetchingQueue %s cannot schedule seek(%s) because prefetch runtime is unavailable",
+ this,
+ request.seekReason)));
+ }
+
+ private void applySeekResetUnderWriteLock(final PendingSeekRequest request) {
+ // 1. Clean up all queued and in-flight events
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ // 2. Discard stale pending entries from in-memory queue
+ pendingEntries.clear();
+
+ // 3. Reset per-writer release state and source-level dedup frontiers.
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ if (Objects.nonNull(request.committedRegionProgress)
+ && !request.committedRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(request.committedRegionProgress);
+ }
+
+ // 4. Reset WAL read position
+ nextExpectedSearchIndex.set(request.targetSearchIndex);
+ requestSubscriptionWalReset(request.targetSearchIndex, seekGeneration.get());
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = seekGeneration.get();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+
+ // 5. Reset commit state to the writer progress immediately before the first re-delivered
+ // entry so seek/rebind resumes from the intended frontier.
+ commitManager.resetState(
+ brokerId, topicName, consensusGroupId, request.committedRegionProgress);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seek({}) applied to searchIndex={}, writerCount={}, seekGeneration={}",
+ this,
+ request.seekReason,
+ request.targetSearchIndex,
+ Objects.nonNull(request.committedRegionProgress)
+ ? request.committedRegionProgress.getWriterPositions().size()
+ : 0,
+ seekGeneration.get());
+ }
+
+ private RegionProgress computeTailRegionProgress() {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final Map<WriterId, WriterProgress> tailProgressByWriter = new LinkedHashMap<>();
+ final File[] walFiles = WALFileUtils.listAllWALFiles(walNode.getLogDirectory());
+ if (Objects.isNull(walFiles) || walFiles.length == 0) {
+ mergeTailProgress(tailProgressByWriter, walNode.getCurrentWALMetaDataSnapshot());
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ WALFileUtils.ascSortByVersionId(walFiles);
+ final long liveVersionId = walNode.getCurrentWALFileVersion();
+ final WALMetaData liveSnapshot = walNode.getCurrentWALMetaDataSnapshot();
+ for (final File walFile : walFiles) {
+ final long versionId = WALFileUtils.parseVersionId(walFile.getName());
+ if (versionId == liveVersionId) {
+ mergeTailProgress(tailProgressByWriter, liveSnapshot);
+ continue;
+ }
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ mergeTailProgress(tailProgressByWriter, reader.getMetaData());
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to read WAL metadata from {} while computing seekToEnd frontier",
+ this,
+ walFile,
+ e);
+ }
+ }
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ private void mergeTailProgress(
+ final Map<WriterId, WriterProgress> tailProgressByWriter, final WALMetaData metadata) {
+ if (Objects.isNull(metadata)) {
+ return;
+ }
+ final List<Long> physicalTimes = metadata.getPhysicalTimes();
+ final List<Short> nodeIds = metadata.getNodeIds();
+ final List<Long> localSeqs = metadata.getLocalSeqs();
+ final int size = Math.min(Math.min(physicalTimes.size(), nodeIds.size()), localSeqs.size());
+ for (int i = 0; i < size; i++) {
+ final int writerNodeId = nodeIds.get(i);
+ final long physicalTime = physicalTimes.get(i);
+ final long localSeq = localSeqs.get(i);
+ if (writerNodeId < 0 || physicalTime < 0L || localSeq < 0L) {
+ continue;
+ }
+
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), writerNodeId);
+ final WriterProgress candidateProgress = new WriterProgress(physicalTime, localSeq);
+ final WriterProgress currentProgress = tailProgressByWriter.get(writerId);
+ if (Objects.isNull(currentProgress)
+ || compareWriterProgress(candidateProgress, currentProgress) > 0) {
+ tailProgressByWriter.put(writerId, candidateProgress);
+ }
+ }
+ }
+
+ /**
+ * Extracts the maximum timestamp from an InsertNode. For row nodes this is the single timestamp;
+ * for tablet nodes, {@code times} is sorted so the last element is the max. For composite nodes,
+ * iterates over children.
+ *
+ * @return the maximum timestamp, or {@code Long.MIN_VALUE} if extraction fails
+ */
+ private long extractMaxTime(final InsertNode insertNode) {
+ try {
+ if (insertNode instanceof InsertRowNode) {
+ return ((InsertRowNode) insertNode).getTime();
+ }
+ if (insertNode instanceof InsertTabletNode) {
+ final InsertTabletNode tabletNode = (InsertTabletNode) insertNode;
+ final int rowCount = tabletNode.getRowCount();
+ return rowCount > 0 ? tabletNode.getTimes()[rowCount - 1] : Long.MIN_VALUE;
+ }
+ if (insertNode instanceof InsertMultiTabletsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertTabletNode child :
+ ((InsertMultiTabletsNode) insertNode).getInsertTabletNodeList()) {
+ final int rowCount = child.getRowCount();
+ if (rowCount > 0) {
+ max = Math.max(max, child.getTimes()[rowCount - 1]);
+ }
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row : ((InsertRowsNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsOfOneDeviceNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row :
+ ((InsertRowsOfOneDeviceNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ // Fallback: use getMinTime() which at least gets a timestamp
+ return insertNode.getMinTime();
+ } catch (final Exception e) {
+ return Long.MIN_VALUE;
+ }
+ }
+
+ /**
+ * Checks whether it is time to inject a watermark event and does so if the configured interval
+ * has elapsed. Called from prefetch rounds after processing data and during idle scheduling.
+ */
+ private void maybeInjectWatermark() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return; // No data observed yet, nothing to report
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0) {
+ return; // Watermark disabled
+ }
+ final long now = System.currentTimeMillis();
+ if (now - lastWatermarkEmitTimeMs >= intervalMs) {
+ injectWatermark(maxObservedTimestamp);
+ lastWatermarkEmitTimeMs = now;
+ }
+ }
+
+ /**
+ * Injects a {@link SubscriptionPollResponseType#WATERMARK} event into the prefetching queue. The
+ * committed mapping is deliberately NOT recorded because watermark events are metadata, not user
+ * data.
+ *
+ * @param watermarkTimestamp the maximum data timestamp observed so far
+ */
+ private void injectWatermark(final long watermarkTimestamp) {
+ final int dataNodeId = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final SubscriptionCommitContext watermarkCtx = createNonCommittableSeekContext(dataNodeId);
+ final SubscriptionEvent watermarkEvent =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.WATERMARK.getType(),
+ new WatermarkPayload(watermarkTimestamp, dataNodeId),
+ watermarkCtx);
+ prefetchingQueue.add(watermarkEvent);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: injected WATERMARK, watermarkTimestamp={}",
+ this,
+ watermarkTimestamp);
+ }
+
+ /** Returns the maximum observed data timestamp for metrics. */
+ public long getMaxObservedTimestamp() {
+ return maxObservedTimestamp;
+ }
+
+ private void markAcceptedFromPending() {
+ pendingPathAcceptedEntries.incrementAndGet();
+ }
+
+ private void markAcceptedFromWal() {
+ walPathAcceptedEntries.incrementAndGet();
+ }
+
+ public void close() {
+ final PendingSeekRequest seekRequestToFail;
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> prefetchBinding;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ closeRequested = true;
+ seekRequestToFail = pendingSeekRequest;
+ pendingSeekRequest = null;
+ } finally {
+ releaseWriteLock();
+ }
+
+ prefetchBinding = detachPrefetchSubtask();
+
+ if (Objects.nonNull(seekRequestToFail)) {
+ seekRequestToFail.fail(
+ new IllegalStateException(
+ String.format("ConsensusPrefetchingQueue %s is closing before seek applies", this)));
+ }
+
+ if (Objects.nonNull(prefetchBinding.right)) {
+ prefetchBinding.right.cancelPendingExecution();
+ prefetchBinding.right.awaitIdle();
+ }
+
+ try {
+ acquireWriteLock();
+ try {
+ if (!isClosed
+ && pendingSeekRequest == null
+ && seekGeneration.get() == observedSeekGeneration) {
+ flushLingeringBatchOnCloseUnderWriteLock();
+ }
+ markClosed();
+ } finally {
+ releaseWriteLock();
+ }
+
+ // Deregister metrics after the queue is fully closed.
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance()
+ .deregister(getPrefetchingQueueId());
+
+ if (Objects.nonNull(prefetchBinding.left) && Objects.nonNull(prefetchBinding.right)) {
+ if (!prefetchBinding.left.isShutdown()) {
+ prefetchBinding.left.deregister(prefetchBinding.right.getTaskId());
+ } else {
+ prefetchBinding.right.close();
+ }
+ }
+
+ try {
+ // Deregister from IoTConsensusServerImpl (stop receiving in-memory data).
+ serverImpl.deregisterSubscriptionQueue(pendingEntries);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error during deregister", this, e);
+ } finally {
+ try {
+ cleanUp();
+ } finally {
+ // Persist progress before closing
+ commitManager.persistAll();
+ }
+ }
+ } finally {
+ closeRequested = false;
+ }
+ }
+
+ private void flushLingeringBatchOnCloseUnderWriteLock() {
+ if (lingerBatch.isEmpty()) {
+ return;
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: flushing {} lingering tablets during close",
+ this,
+ lingerBatch.tablets.size());
+ if (!flushBatch(lingerBatch, observedSeekGeneration)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to flush lingering batch during close, discarding it",
+ this);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ }
+ }
+
+ private SubscriptionEvent generateErrorResponse(final String errorMessage) {
+ return new SubscriptionEvent(
+ SubscriptionPollResponseType.ERROR.getType(),
+ new ErrorPayload(errorMessage, false),
+ createNonCommittableContext(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()));
+ }
+
+ private SubscriptionEvent generateOutdatedErrorResponse() {
+ return new SubscriptionEvent(
+ SubscriptionPollResponseType.ERROR.getType(),
+ ErrorPayload.OUTDATED_ERROR_PAYLOAD,
+ createNonCommittableContext(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()));
+ }
+
+ /**
+ * Shared subscription events still use {@link SubscriptionCommitContext#INVALID_COMMIT_ID} to
+ * mark metadata and error payloads as non-committable. Consensus correctness never treats this
+ * sentinel as a replay or commit frontier.
+ */
+ private SubscriptionCommitContext createNonCommittableContext(final int dataNodeId) {
+ return new SubscriptionCommitContext(
+ dataNodeId,
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ INVALID_COMMIT_ID);
+ }
+
+ private SubscriptionCommitContext createNonCommittableSeekContext(final int dataNodeId) {
+ return new SubscriptionCommitContext(
+ dataNodeId,
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ INVALID_COMMIT_ID,
+ seekGeneration.get(),
+ consensusGroupId.toString());
+ }
+
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ return PipeDataNodeAgent.runtime().getRebootTimes() > commitContext.getRebootTimes()
+ || seekGeneration.get() != commitContext.getSeekGeneration();
+ }
+
+ // ======================== Status ========================
+
+ public boolean isClosed() {
+ return isClosed;
+ }
+
+ public void markClosed() {
+ isClosed = true;
+ }
+
+ // ======================== Routing Runtime Version Control ========================
+
+ public long getWalGapSkippedEntries() {
+ return walGapSkippedEntries.get();
+ }
+
+ public long getEpochChangeCount() {
+ return runtimeVersionChangeCount.get();
+ }
+
+ // ======================== Leader Activation ========================
+
+ /**
+ * Activates or deactivates this queue. Only the preferred-writer (leader) node's queue should be
+ * active. Inactive queues skip prefetching and return null on poll.
+ */
+ public void setActive(final boolean active) {
+ this.isActive = active;
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: isActive set to {} (region={})",
+ this,
+ active,
+ consensusGroupId);
+ if (active) {
+ requestPrefetch();
+ }
+ }
+
+ public boolean isActive() {
+ return isActive;
+ }
+
+ public void setActiveWriterNodeIds(final Set<Integer> activeWriterNodeIds) {
+ this.runtimeActiveWriterNodeIds =
+ Collections.unmodifiableSet(
+ new LinkedHashSet<>(Objects.requireNonNull(activeWriterNodeIds)));
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: runtimeActiveWriterNodeIds={}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, orderMode={}, preferredWriterNodeId={})",
+ this,
+ this.runtimeActiveWriterNodeIds,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ orderMode,
+ preferredWriterNodeId);
+ requestPrefetch();
+ }
+
+ private void refreshEffectiveActiveWriterNodeIds() {
+ final LinkedHashSet<Integer> effectiveWriterNodeIds = new LinkedHashSet<>();
+ switch (orderMode) {
+ case TopicConstant.ORDER_MODE_MULTI_WRITER_VALUE:
+ effectiveWriterNodeIds.addAll(runtimeActiveWriterNodeIds);
+ if (effectiveWriterNodeIds.isEmpty() && preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ break;
+ case TopicConstant.ORDER_MODE_PER_WRITER_VALUE:
+ if (preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ break;
+ case TopicConstant.ORDER_MODE_LEADER_ONLY_VALUE:
+ default:
+ if (preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ if (previousPreferredWriterNodeId >= 0
+ && previousPreferredWriterNodeId != preferredWriterNodeId
+ && runtimeActiveWriterNodeIds.contains(previousPreferredWriterNodeId)) {
+ effectiveWriterNodeIds.add(previousPreferredWriterNodeId);
+ }
+ break;
+ }
+ this.activeWriterNodeIds = Collections.unmodifiableSet(effectiveWriterNodeIds);
+ }
+
+ public void setPreferredWriterNodeId(final int preferredWriterNodeId) {
+ if (this.preferredWriterNodeId != preferredWriterNodeId) {
+ previousPreferredWriterNodeId = this.preferredWriterNodeId;
+ } else {
+ previousPreferredWriterNodeId = -1;
+ }
+ this.preferredWriterNodeId = preferredWriterNodeId;
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: preferredWriterNodeId set to {}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, orderMode={})",
+ this,
+ this.preferredWriterNodeId,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ orderMode);
+ requestPrefetch();
+ }
+
+ public Set<Integer> getActiveWriterNodeIds() {
+ return activeWriterNodeIds;
+ }
+
+ public void setOrderMode(final String orderMode) {
+ final String normalizedOrderMode = TopicConfig.normalizeOrderMode(orderMode);
+ if (Objects.equals(this.orderMode, normalizedOrderMode)) {
+ return;
+ }
+ this.orderMode = normalizedOrderMode;
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: orderMode set to {}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, preferredWriterNodeId={}, runtimeActiveWriterNodeIds={})",
+ this,
+ this.orderMode,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ preferredWriterNodeId,
+ runtimeActiveWriterNodeIds);
+ requestPrefetch();
+ }
+
+ public String getOrderMode() {
+ return orderMode;
+ }
+
+ private boolean isLaneRuntimeActive(final WriterLaneId laneId) {
+ final Set<Integer> writerNodeIds = activeWriterNodeIds;
+ return writerNodeIds.isEmpty() || writerNodeIds.contains(laneId.writerNodeId);
+ }
+
+ public void applyRuntimeState(final ConsensusRegionRuntimeState runtimeState) {
+ Objects.requireNonNull(runtimeState, "runtimeState");
+ this.runtimeVersion = runtimeState.getRuntimeVersion();
+ runtimeVersionChangeCount.incrementAndGet();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: applied runtimeVersion {}",
+ this,
+ runtimeState.getRuntimeVersion());
+ setPreferredWriterNodeId(runtimeState.getPreferredWriterNodeId());
+ setActiveWriterNodeIds(runtimeState.getActiveWriterNodeIds());
+ // "active" decides whether this replica should serve subscription traffic on the current node.
+ // In multi-writer mode, activeWriterNodeIds may intentionally include follower replicas for
+ // ordering/watermark coordination, so it must not be reused as the local service-activation
+ // signal.
+ setActive(runtimeState.isActive());
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: applied runtimeState={}, preferredWriterNodeId={}",
+ this,
+ runtimeState,
+ runtimeState.getPreferredWriterNodeId());
+ if (runtimeState.isActive()) {
+ requestPrefetch();
+ }
+ }
+
+ public String getPrefetchingQueueId() {
+ return brokerId + "_" + topicName;
+ }
+
+ public long getSubscriptionUncommittedEventCount() {
+ return inFlightEvents.size();
+ }
+
+ /** Exposes the current seek generation for runtime tests and metrics. */
+ public long getCurrentSeekGeneration() {
+ return seekGeneration.get();
+ }
+
+ public int getPrefetchedEventCount() {
+ return prefetchingQueue.size();
+ }
+
+ public long getCurrentReadSearchIndex() {
+ return nextExpectedSearchIndex.get();
+ }
+
+ public long getPendingPathAcceptedEntries() {
+ return pendingPathAcceptedEntries.get();
+ }
+
+ public long getWalPathAcceptedEntries() {
+ return walPathAcceptedEntries.get();
+ }
+
+ public String getBrokerId() {
+ return brokerId;
+ }
+
+ public String getTopicName() {
+ return topicName;
+ }
+
+ public ConsensusGroupId getConsensusGroupId() {
+ return consensusGroupId;
+ }
+
+ /**
+ * Returns an approximate backlog for this queue.
+ *
+ * <p>The metric intentionally avoids collapsing per-writer committed progress into a single
+ * scalar local sequence. Instead it counts queued/in-flight work and adds one extra unit when the
+ * local WAL reader still has unread entries beyond its current replay cursor.
+ */
+ public long getLag() {
+ long lag =
+ prefetchingQueue.size()
+ + inFlightEvents.size()
+ + pendingEntries.size()
+ + getRealtimeBufferedEntryCount();
+ if (nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex()) {
+ lag++;
+ }
+ return lag;
+ }
+
+ // ======================== Stringify ========================
+
+ public Map<String, String> coreReportMessage() {
+ final Map<String, String> result = new HashMap<>();
+ result.put("brokerId", brokerId);
+ result.put("topicName", topicName);
+ result.put("consensusGroupId", consensusGroupId.toString());
+ result.put("currentReadSearchIndex", String.valueOf(nextExpectedSearchIndex.get()));
+ result.put("prefetchingQueueSize", String.valueOf(prefetchingQueue.size()));
+ result.put("inFlightEventsSize", String.valueOf(inFlightEvents.size()));
+ result.put("pendingEntriesSize", String.valueOf(pendingEntries.size()));
+ result.put("pendingPathAcceptedEntries", String.valueOf(getPendingPathAcceptedEntries()));
+ result.put("walPathAcceptedEntries", String.valueOf(getWalPathAcceptedEntries()));
+ result.put("seekGeneration", String.valueOf(seekGeneration.get()));
+ result.put("walGapSkippedEntries", String.valueOf(walGapSkippedEntries.get()));
+ result.put("bufferedRealtimeEntryCount", String.valueOf(getRealtimeBufferedEntryCount()));
+ result.put("lag", String.valueOf(getLag()));
+ result.put("isClosed", String.valueOf(isClosed));
+ result.put("isActive", String.valueOf(isActive));
+ result.put("orderMode", orderMode);
+ result.put("preferredWriterNodeId", String.valueOf(preferredWriterNodeId));
+ result.put("activeWriterCount", String.valueOf(activeWriterNodeIds.size()));
+ result.put("runtimeActiveWriterCount", String.valueOf(runtimeActiveWriterNodeIds.size()));
+ result.put("recoveryWriterCount", String.valueOf(recoveryWriterProgressByWriter.size()));
+ result.put("writerLaneCount", String.valueOf(writerLanes.size()));
+ result.put("realtimeLaneCount", String.valueOf(realtimeEntriesByLane.size()));
+ return result;
+ }
+
+ @Override
+ public String toString() {
+ return "ConsensusPrefetchingQueue" + coreReportMessage();
+ }
+
+ // ======================== Inner Classes ========================
+
+ private interface LaneBufferedEntry {
+ List<Tablet> getTablets();
+
+ long getPhysicalTime();
+
+ int getWriterNodeId();
+
+ long getLocalSeq();
+
+ long getSearchIndex();
+
+ OrderingKey getOrderingKey();
+ }
+
+ private static final class DeliveryBatchState {
+
+ private final List<Tablet> tablets = new ArrayList<>();
+ private long startSearchIndex;
+ private long endSearchIndex;
+ private long estimatedBytes;
+ private long firstTabletTimeMs;
+ private long physicalTime;
+ private int writerNodeId;
+ private long lastLocalSeq;
+ private int entryCount;
+
+ private DeliveryBatchState() {
+ reset();
+ }
+
+ private boolean isEmpty() {
+ return tablets.isEmpty();
+ }
+
+ private void append(
+ final LaneBufferedEntry entry,
+ final long entryEstimatedBytes,
+ final boolean trackLingerTime) {
+ if (tablets.isEmpty()) {
+ if (trackLingerTime) {
+ firstTabletTimeMs = System.currentTimeMillis();
+ }
+ writerNodeId = entry.getWriterNodeId();
+ }
+ if (entry.getSearchIndex() >= 0) {
+ if (startSearchIndex < 0) {
+ startSearchIndex = entry.getSearchIndex();
+ }
+ endSearchIndex = entry.getSearchIndex();
+ }
+ tablets.addAll(entry.getTablets());
+ estimatedBytes += entryEstimatedBytes;
+ physicalTime = entry.getPhysicalTime();
+ lastLocalSeq = entry.getLocalSeq();
+ writerNodeId = entry.getWriterNodeId();
+ entryCount++;
+ }
+
+ private void reset() {
+ tablets.clear();
+ startSearchIndex = -1L;
+ endSearchIndex = -1L;
+ estimatedBytes = 0L;
+ firstTabletTimeMs = 0L;
+ physicalTime = 0L;
+ lastLocalSeq = -1L;
+ writerNodeId = -1;
+ entryCount = 0;
+ }
+ }
+
+ private static final class WriterLaneId {
+ private final int writerNodeId;
+
+ private WriterLaneId(final int writerNodeId) {
+ this.writerNodeId = writerNodeId;
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if (this == obj) {
+ return true;
+ }
+ if (!(obj instanceof WriterLaneId)) {
+ return false;
+ }
+ final WriterLaneId that = (WriterLaneId) obj;
+ return writerNodeId == that.writerNodeId;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(writerNodeId);
+ }
Review Comment:
now use writerNodeId directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3352929737
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -731,22 +728,19 @@ public void updateCompletedTopicNames(final String consumerGroupId, final String
}
public void unbindPrefetchingQueue(final String consumerGroupId, final String topicName) {
- // Try consensus broker first
- final ConsensusSubscriptionBroker consensusBroker =
- consumerGroupIdToConsensusBroker.get(consumerGroupId);
- if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
- consensusBroker.unbindConsensusPrefetchingQueue(topicName);
- prefetchingQueueCount.invalidate();
- return;
- }
- // Fall back to pipe broker
- final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
- if (Objects.isNull(pipeBroker)) {
+ final ISubscriptionBroker broker = getBrokerForTopic(consumerGroupId, topicName);
+ if (Objects.isNull(broker)) {
LOGGER.warn(
"Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
return;
}
- pipeBroker.unbindPrefetchingQueue(topicName);
+ if (broker instanceof ConsensusSubscriptionBroker) {
+ ((ConsensusSubscriptionBroker) broker).unbindConsensusPrefetchingQueue(topicName);
+ } else if (broker instanceof SubscriptionBroker) {
+ ((SubscriptionBroker) broker).unbindPrefetchingQueue(topicName);
+ } else {
+ broker.removeQueue(topicName);
+ }
Review Comment:
Abstract as `ISubscriptionBroker.unbind(String topicName)` ?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -797,23 +786,27 @@ public int getPrefetchingQueueCount() {
public Map<String, Long> getConsensusLagSummary() {
final Map<String, Long> result = new ConcurrentHashMap<>();
- for (final Map.Entry<String, ConsensusSubscriptionBroker> entry :
- consumerGroupIdToConsensusBroker.entrySet()) {
- final String groupId = entry.getKey();
- for (final Map.Entry<String, Long> lag : entry.getValue().getLagSummary().entrySet()) {
- result.put(groupId + "/" + lag.getKey(), lag.getValue());
+ for (final Map.Entry<String, List<ISubscriptionBroker>> entry :
+ consumerGroupIdToBrokers.entrySet()) {
+ for (final ISubscriptionBroker broker : entry.getValue()) {
+ if (!(broker instanceof ConsensusSubscriptionBroker)) {
+ continue;
+ }
+ for (final Map.Entry<String, Long> lag :
+ ((ConsensusSubscriptionBroker) broker).getLagSummary().entrySet()) {
+ result.put(entry.getKey() + "/" + lag.getKey(), lag.getValue());
+ }
}
Review Comment:
May also allow PipeBroker to return an empty summary for now.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
Review Comment:
I mean `boolean onRequest(SearchableRequestMeta request)`;
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -297,17 +297,25 @@ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
}
- // Build column arrays and bitmaps using bulk copy
- final long[] newTimes = Arrays.copyOf(times, rowCount);
+ // Reuse complete tablets on the server-side read path. Column filtering changes tablet shape
+ // and still needs copies for the selected columns.
+ final long[] newTimes = allColumnsMatch ? times : Arrays.copyOf(times, rowCount);
Review Comment:
Why does column filtering need to copy the content of a column?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3108229902
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/subscription/subscription/CreateSubscriptionProcedure.java:
##########
@@ -103,15 +107,41 @@ protected boolean executeFromValidate(final ConfigNodeProcedureEnv env)
alterConsumerGroupProcedure =
new AlterConsumerGroupProcedure(updatedConsumerGroupMeta, subscriptionInfo);
- // Construct CreatePipeProcedureV2s
+ // Construct CreatePipeProcedureV2s (for non-consensus topics)
for (final String topicName : subscribeReq.getTopicNames()) {
+ final TopicMeta topicMeta = subscriptionInfo.get().deepCopyTopicMeta(topicName);
+
+ // Check if this topic should use consensus subscription: mode is live, format is Tablet
+ final String topicMode =
+ topicMeta
+ .getConfig()
+ .getStringOrDefault(TopicConstant.MODE_KEY, TopicConstant.MODE_DEFAULT_VALUE);
+ final String topicFormat =
+ topicMeta
+ .getConfig()
+ .getStringOrDefault(TopicConstant.FORMAT_KEY, TopicConstant.FORMAT_DEFAULT_VALUE);
+ final boolean isConsensusBasedTopic =
+ TopicConstant.MODE_LIVE_VALUE.equalsIgnoreCase(topicMode)
+ && !TopicConstant.FORMAT_TS_FILE_HANDLER_VALUE.equalsIgnoreCase(topicFormat);
Review Comment:
Now the new mode explicitly uses parameter TopicConstant.MODE_KEY (MODE_SNAPSHOT_VALUE / MODE_LIVE_VALUE / MODE_CONSENSUS_VALUE). The new consensus mode will throw an exception if format is not RECORD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3278706848
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionProvider.java:
##########
@@ -338,14 +343,107 @@ Map<String, TopicConfig> unsubscribe(final Set<String> topicNames) throws Subscr
return unsubscribeResp.getTopics();
}
+ void seek(final String topicName, final short seekType, final long timestamp)
+ throws SubscriptionException {
+ final PipeSubscribeSeekReq req;
+ try {
+ req = PipeSubscribeSeekReq.toTPipeSubscribeReq(topicName, seekType, timestamp);
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "IOException occurred when SubscriptionProvider {} serialize seek request for topic {}",
+ this,
+ topicName,
+ e);
+ throw new SubscriptionRuntimeNonCriticalException(e.getMessage(), e);
+ }
+ final TPipeSubscribeResp resp;
+ try {
+ resp = getSessionConnection().pipeSubscribe(req);
Review Comment:
Removed the pipe wording from the new seek wrapper and call-site names. Kept generated thrift names unchanged because they belong to the existing transport protocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279164027
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = inFlightEvents.get(new Pair<>(consumerId, commitContext));
+ if (Objects.isNull(event)) {
+ if (isCommitContextOutdated(commitContext)) {
+ return generateOutdatedErrorResponse();
+ }
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
+ this, consumerId, commitContext));
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ // ======================== Prefetch Round Drive ========================
+
+ private static final long WAL_GAP_RETRY_SLEEP_MS = 10L;
+ private static final long WAL_GAP_WAIT_LOG_INTERVAL_MS = 5_000L;
+
+ private static final long PREFETCH_STATS_LOG_INTERVAL_MS = 5_000L;
+
+ public PrefetchRoundResult drivePrefetchOnce() {
+ if (applyPendingSeekRequestIfNecessary()) {
+ return closeRequested ? PrefetchRoundResult.dormant() : PrefetchRoundResult.rescheduleNow();
+ }
+
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !prefetchInitialized) {
+ return PrefetchRoundResult.dormant();
+ }
+
+ logPeriodicStatsIfNecessary();
+
+ final long currentSeekGeneration = seekGeneration.get();
+ if (currentSeekGeneration != observedSeekGeneration) {
+ resetRoundStateForSeek(currentSeekGeneration);
+ }
+
+ applyPendingSubscriptionWalReset(observedSeekGeneration);
+ recycleInFlightEvents();
+
+ if (!isActive || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return computeIdleRoundResult();
+ }
+
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+ final int batchMaxDelayMs = config.getSubscriptionConsensusBatchMaxDelayInMs();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+
+ final List<IndexedConsensusRequest> batch = drainPendingEntries(maxWalEntries);
+ if (!batch.isEmpty()) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: drained {} entries from pendingEntries, "
+ + "first searchIndex={}, last searchIndex={}, nextExpected={}, "
+ + "prefetchingQueueSize={}",
+ this,
+ batch.size(),
+ batch.get(0).getSearchIndex(),
+ batch.get(batch.size() - 1).getSearchIndex(),
+ nextExpectedSearchIndex.get(),
+ prefetchingQueue.size());
+
+ final boolean batchAccepted =
+ accumulateFromPending(
+ batch, lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes);
+ if (!batchAccepted) {
+ if (pendingWalGapRetryRequested) {
+ // Once a drained batch hits an unresolved WAL gap, the affected suffix falls back to
+ // the WAL path on later rounds instead of being requeued into the bounded pending path.
+ return PrefetchRoundResult.rescheduleAfter(WAL_GAP_RETRY_SLEEP_MS);
+ }
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ }
+
+ if (batch.isEmpty() && lingerBatch.isEmpty()) {
+ tryCatchUpFromWAL(observedSeekGeneration);
+ }
+
+ if (!drainBufferedRealtimeLanes(
+ lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes)) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerElapsedMs = System.currentTimeMillis() - lingerBatch.firstTabletTimeMs;
+ if (lingerElapsedMs >= batchMaxDelayMs) {
+ if (seekGeneration.get() != observedSeekGeneration) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: time-based flush, {} tablets lingered for {}ms "
+ + "(threshold={}ms)",
+ this,
+ lingerBatch.tablets.size(),
+ lingerElapsedMs,
+ batchMaxDelayMs);
+ flushBatch(lingerBatch, observedSeekGeneration);
+ }
+ }
+
+ maybeInjectWatermark();
+ return computeIdleRoundResult();
+ } catch (final Throwable fatal) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: prefetch round failed " + "(type={}, message={})",
+ this,
+ fatal.getClass().getName(),
+ fatal.getMessage(),
+ fatal);
+ if (fatal instanceof VirtualMachineError) {
+ markClosed();
+ return PrefetchRoundResult.dormant();
+ }
+ return PrefetchRoundResult.rescheduleAfter(100L);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private void logPeriodicStatsIfNecessary() {
+ final long nowMs = System.currentTimeMillis();
+ if (nowMs - lastStatsLogTimeMs < PREFETCH_STATS_LOG_INTERVAL_MS) {
+ return;
+ }
+
+ final long currentPendingAcceptedEntries = pendingPathAcceptedEntries.get();
+ final long currentWalAcceptedEntries = walPathAcceptedEntries.get();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: periodic stats, lag={}, pendingDelta={}, walDelta={}, "
+ + "pendingTotal={}, walTotal={}, pendingQueueSize={}, prefetchingQueueSize={}, "
+ + "inFlightEventsSize={}, realtimeLaneCount={}, walHasNext={}, isActive={}, subtaskScheduled={}",
+ this,
+ getLag(),
+ currentPendingAcceptedEntries - lastPendingAcceptedEntries,
+ currentWalAcceptedEntries - lastWalAcceptedEntries,
+ currentPendingAcceptedEntries,
+ currentWalAcceptedEntries,
+ pendingEntries.size(),
+ prefetchingQueue.size(),
+ inFlightEvents.size(),
+ realtimeEntriesByLane.size(),
+ hasReadableWalEntries(),
+ isActive,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ lastStatsLogTimeMs = nowMs;
+ lastPendingAcceptedEntries = currentPendingAcceptedEntries;
+ lastWalAcceptedEntries = currentWalAcceptedEntries;
+ }
+
+ private void resetRoundStateForSeek(final long newSeekGeneration) {
+ restorePendingSubscriptionWalCursor(newSeekGeneration);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = newSeekGeneration;
+ }
+
+ private List<IndexedConsensusRequest> drainPendingEntries(final int maxWalEntries) {
+ final List<IndexedConsensusRequest> batch = new ArrayList<>();
+ IndexedConsensusRequest next;
+ while (batch.size() < maxWalEntries && (next = pendingEntries.poll()) != null) {
+ batch.add(next);
+ }
+ return batch;
+ }
+
+ private PrefetchRoundResult computeIdleRoundResult() {
+ if (isClosed || !prefetchInitialized || !isActive) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (hasImmediatePrefetchableWork()) {
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ long delayMs = Long.MAX_VALUE;
+ if (hasHistoricalWalLag()) {
+ delayMs = Math.min(delayMs, WAL_GAP_RETRY_SLEEP_MS);
+ }
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerDelayMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusBatchMaxDelayInMs()
+ - (System.currentTimeMillis() - lingerBatch.firstTabletTimeMs);
+ delayMs = Math.min(delayMs, Math.max(1L, lingerDelayMs));
+ }
+
+ final long watermarkDelayMs = computeWatermarkDelayMs();
+ if (watermarkDelayMs > 0L) {
+ delayMs = Math.min(delayMs, watermarkDelayMs);
+ }
+
+ if (!inFlightEvents.isEmpty()) {
+ delayMs =
+ Math.min(
+ delayMs,
+ SubscriptionConfig.getInstance().getSubscriptionRecycleUncommittedEventIntervalMs());
+ }
+
+ return delayMs == Long.MAX_VALUE
+ ? PrefetchRoundResult.dormant()
+ : PrefetchRoundResult.rescheduleAfter(delayMs);
+ }
+
+ private long computeWatermarkDelayMs() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return -1L;
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0L) {
+ return -1L;
+ }
+ if (lastWatermarkEmitTimeMs == 0L) {
+ return 1L;
+ }
+ final long elapsedMs = System.currentTimeMillis() - lastWatermarkEmitTimeMs;
+ return elapsedMs >= intervalMs ? 1L : Math.max(1L, intervalMs - elapsedMs);
+ }
+
+ private boolean hasImmediatePrefetchableWork() {
+ return !pendingEntries.isEmpty() || !realtimeEntriesByLane.isEmpty() || hasReadableWalEntries();
+ }
+
+ private boolean hasHistoricalWalLag() {
+ return nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex();
+ }
+
+ /**
+ * Accumulates tablets from pending entries into the linger buffer. When pending replay outruns
+ * the local WAL reader, this method backfills the local-index gap from WAL before continuing.
+ *
+ * @return false if the batch became stale because seek generation changed while flushing
+ */
+ private static boolean hasLocalSearchIndex(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() >= 0;
+ }
Review Comment:
moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #17238:
URL: https://github.com/apache/iotdb/pull/17238
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3353739260
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
Review Comment:
Done in 75bacb1. Added return-value javadoc directly to `SearchableRequestVisitor.onRequest(SearchableRequestMeta request)` explaining that true continues scanning and false stops immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3108226564
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java:
##########
@@ -389,6 +389,13 @@ public class CommonConfig {
private long subscriptionMetaSyncerInitialSyncDelayMinutes = 3;
private long subscriptionMetaSyncerSyncIntervalMinutes = 3;
+ private int subscriptionConsensusBatchMaxDelayInMs = 50;
+ private long subscriptionConsensusBatchMaxSizeInBytes = 8 * MB;
+ private int subscriptionConsensusBatchMaxTabletCount = 64;
+ private int subscriptionConsensusBatchMaxWalEntries = 128;
+
+ private long subscriptionConsensusWalRetentionSizeInBytes = 512 * MB;
Review Comment:
The scope of `throttleThreshold` is all WALNodes of one DataNode, but the scope of retention size is per region. As the number of region changes, we can't directly use (`throttleThreshold` / region count) for it. The default value of retention policy can be discussed later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3278703156
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionConsumer.java:
##########
@@ -1395,6 +1633,365 @@ private void unsubscribeWithRedirection(final Set<String> topicNames)
throw new SubscriptionRuntimeCriticalException(errorMessage);
}
+ /**
+ * Sends seek request to ALL available providers. Unlike subscribe/unsubscribe, seek is only
+ * considered successful if every available provider acknowledges it because data regions for the
+ * topic may be distributed across different nodes.
+ */
+ private void seekWithRedirection(
+ final String topicName, final short seekType, final long timestamp)
+ throws SubscriptionException {
Review Comment:
Renamed to seekOnAllProviders
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279117542
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/node/WALNode.java:
##########
@@ -898,11 +925,117 @@ public long getCurrentWALFileVersion() {
return buffer.getCurrentWALFileVersion();
}
+ public WALMetaData getCurrentWALMetaDataSnapshot() {
+ return buffer.getCurrentWALMetaDataSnapshot();
+ }
+
@Override
public long getTotalSize() {
return WALManager.getInstance().getTotalDiskUsage();
}
+ @Override
+ public long getRegionDiskUsage() {
+ return buffer.getDiskUsage();
+ }
+
+ @Override
+ public long getSearchIndexToFreeAtLeast(long bytesToFree) {
+ if (bytesToFree <= 0) {
+ return DEFAULT_SAFELY_DELETED_SEARCH_INDEX;
+ }
+ File[] walFiles = WALFileUtils.listAllWALFiles(logDirectory);
+ if (walFiles == null || walFiles.length <= 1) {
+ // No files or only the current-writing file — cannot free anything
+ return DEFAULT_SAFELY_DELETED_SEARCH_INDEX;
+ }
+ WALFileUtils.ascSortByVersionId(walFiles);
+ // Exclude the last file (currently being written)
+ long accumulated = 0;
+ for (int i = 0; i < walFiles.length - 1; i++) {
+ accumulated += walFiles[i].length();
+ if (accumulated >= bytesToFree) {
+ // The next file's startSearchIndex is the boundary: everything before it can be deleted
+ if (i + 1 < walFiles.length) {
+ return WALFileUtils.parseStartSearchIndex(walFiles[i + 1].getName());
+ }
+ break;
+ }
+ }
+ // Could not free enough even by deleting all non-current files — allow deleting all
+ return Long.MAX_VALUE;
+ }
+
+ @Override
+ public long getVersionIdToFreeAtLeast(long bytesToFree) {
Review Comment:
Added combined deletion-bound APIs so size/time retention can get searchIndex and versionId in one WAL file scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279206769
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/ConsensusSubscriptionBroker.java:
##########
@@ -0,0 +1,811 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusLogToTabletConverter;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusPrefetchingQueue;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusRegionRuntimeState;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusSubscriptionCommitManager;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.TopicProgress;
+import org.apache.iotdb.rpc.subscription.payload.request.PipeSubscribeSeekReq;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Collectors;
+
+/**
+ * Consensus-based subscription broker that reads data directly from IoTConsensus WAL. Each instance
+ * manages consensus prefetching queues for a single consumer group.
+ */
+public class ConsensusSubscriptionBroker implements ISubscriptionBroker {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusSubscriptionBroker.class);
+
+ private final String brokerId; // consumer group id
+
+ /** Maps topic name to a list of ConsensusPrefetchingQueues, one per data region. */
+ private final Map<String, List<ConsensusPrefetchingQueue>> topicNameToConsensusPrefetchingQueues;
+
+ /** Round-robin counter for fair polling among region queues already assigned to this consumer. */
+ private final AtomicInteger pollRoundRobinIndex = new AtomicInteger(0);
+
+ private final Map<String, ConcurrentHashMap<String, Long>> topicConsumerLastPollMs =
+ new ConcurrentHashMap<>();
+
+ private final Map<String, TopicOwnershipSnapshot> topicOwnershipSnapshots =
+ new ConcurrentHashMap<>();
+
+ public ConsensusSubscriptionBroker(final String brokerId) {
+ this.brokerId = brokerId;
+ this.topicNameToConsensusPrefetchingQueues = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return topicNameToConsensusPrefetchingQueues.isEmpty();
+ }
+
+ @Override
+ public boolean hasQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ return Objects.nonNull(queues)
+ && !queues.isEmpty()
+ && queues.stream().anyMatch(q -> !q.isClosed());
+ }
+
+ //////////////////////////// poll ////////////////////////////
+
+ @Override
+ public List<SubscriptionEvent> poll(
+ final String consumerId, final Set<String> topicNames, final long maxBytes) {
+ return poll(consumerId, topicNames, maxBytes, Collections.emptyMap());
+ }
+
+ public List<SubscriptionEvent> poll(
+ final String consumerId,
+ final Set<String> topicNames,
+ final long maxBytes,
+ final Map<String, TopicProgress> progressByTopic) {
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll called, consumerId={}, topicNames={}, "
+ + "queueCount={}, maxBytes={}",
+ brokerId,
+ consumerId,
+ topicNames,
+ topicNameToConsensusPrefetchingQueues.size(),
+ maxBytes);
+
+ final List<SubscriptionEvent> eventsToPoll = new ArrayList<>();
+ final List<SubscriptionEvent> eventsToNack = new ArrayList<>();
+ long totalSize = 0;
+
+ for (final String topicName : topicNames) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ continue;
+ }
+
+ final TopicOwnershipSnapshot ownershipSnapshot =
+ refreshAndGetTopicOwnership(topicName, queues, consumerId);
+ final List<ConsensusPrefetchingQueue> assignedQueues =
+ getAssignedQueues(queues, consumerId, ownershipSnapshot);
+ if (assignedQueues.isEmpty()) {
+ continue;
+ }
+
+ final List<ConsensusPrefetchingQueue> pollQueues =
+ buildPollOrderForAssignedQueues(assignedQueues, topicName);
+ final int eventsBeforeTopicPoll = eventsToPoll.size();
+
+ for (final ConsensusPrefetchingQueue consensusQueue : pollQueues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+
+ final String regionIdStr = consensusQueue.getConsensusGroupId().toString();
+ final TopicProgress topicProgress = progressByTopic.get(topicName);
+ final RegionProgress regionProgress =
+ Objects.nonNull(topicProgress)
+ ? topicProgress.getRegionProgress().get(regionIdStr)
+ : null;
+
+ final SubscriptionEvent event = consensusQueue.poll(consumerId, regionProgress);
+ if (Objects.isNull(event)) {
+ continue;
+ }
+
+ final long currentSize;
+ try {
+ currentSize = event.getCurrentResponseSize();
+ } catch (final IOException e) {
+ eventsToNack.add(event);
+ continue;
+ }
+
+ eventsToPoll.add(event);
+ totalSize += currentSize;
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ // Nack any events that had errors
+ if (!eventsToNack.isEmpty()) {
+ commit(
+ consumerId,
+ eventsToNack.stream()
+ .map(SubscriptionEvent::getCommitContext)
+ .collect(Collectors.toList()),
+ true);
+ }
+
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll result, consumerId={}, eventsPolled={}, eventsNacked={}",
+ brokerId,
+ consumerId,
+ eventsToPoll.size(),
+ eventsToNack.size());
+
+ return eventsToPoll;
+ }
+
+ @Override
+ public List<SubscriptionEvent> pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final ConsensusPrefetchingQueue assignedQueue =
+ getAssignedQueueForConsumer(
+ queues, topicName, consumerId, commitContext.getRegionId(), "pollTablets");
+ if (Objects.isNull(assignedQueue)) {
+ return Collections.emptyList();
+ }
+
+ final SubscriptionEvent event = assignedQueue.pollTablets(consumerId, commitContext, offset);
+ if (Objects.nonNull(event)) {
+ return Collections.singletonList(event);
+ }
+ return Collections.emptyList();
+ }
+
+ //////////////////////////// commit ////////////////////////////
+
+ @Override
+ public List<SubscriptionCommitContext> commit(
+ final String consumerId,
+ final List<SubscriptionCommitContext> commitContexts,
+ final boolean nack) {
+ final List<SubscriptionCommitContext> successfulCommitContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext commitContext : commitContexts) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to commit",
+ brokerId,
+ topicName);
+ continue;
+ }
+
+ final ConsensusPrefetchingQueue assignedQueue =
+ getAssignedQueueForConsumer(
+ queues, topicName, consumerId, commitContext.getRegionId(), nack ? "nack" : "ack");
+ boolean handled = false;
+ if (Objects.nonNull(assignedQueue)) {
+ final boolean success;
+ if (!nack) {
+ success = assignedQueue.ackSilent(consumerId, commitContext);
+ } else {
+ success = assignedQueue.nackSilent(consumerId, commitContext);
+ }
+ if (success) {
+ successfulCommitContexts.add(commitContext);
+ handled = true;
+ }
+ }
+ if (!handled) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: commit context {} not found in any of {} region queue(s) for topic [{}]",
+ brokerId,
+ commitContext,
+ queues.size(),
+ topicName);
+ }
+ }
+ return successfulCommitContexts;
+ }
+
+ @Override
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return true;
+ }
+ // Route directly to the correct region queue using regionId
+ final String regionId = commitContext.getRegionId();
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!regionId.isEmpty() && !regionId.equals(q.getConsensusGroupId().toString())) {
+ continue;
+ }
+ return q.isCommitContextOutdated(commitContext);
+ }
+ return true;
+ }
+
+ //////////////////////////// seek ////////////////////////////
+
+ public void seek(final String topicName, final short seekType) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to seek",
+ brokerId,
+ topicName);
+ return;
+ }
+
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ switch (seekType) {
+ case PipeSubscribeSeekReq.SEEK_TO_BEGINNING:
+ queue.seekToBeginning();
+ break;
+ case PipeSubscribeSeekReq.SEEK_TO_END:
+ queue.seekToEnd();
+ break;
+ default:
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: unsupported seekType {} for topic [{}]",
+ brokerId,
+ seekType,
+ topicName);
+ break;
+ }
+ }
+ }
+
+ public void seek(final String topicName, final TopicProgress topicProgress) {
+ final TopicProgress safeProgress =
+ topicProgress != null ? topicProgress : new TopicProgress(Collections.emptyMap());
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to seek(topicProgress)",
+ brokerId,
+ topicName);
+ return;
+ }
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ final RegionProgress regionProgress =
+ safeProgress.getRegionProgress().get(queue.getConsensusGroupId().toString());
+ seekQueueToRegionProgress(queue, regionProgress, false);
+ }
+ }
+
+ public void seekAfter(final String topicName, final TopicProgress topicProgress) {
+ final TopicProgress safeProgress =
+ topicProgress != null ? topicProgress : new TopicProgress(Collections.emptyMap());
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to seekAfter(topicProgress)",
+ brokerId,
+ topicName);
+ return;
+ }
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ final RegionProgress regionProgress =
+ safeProgress.getRegionProgress().get(queue.getConsensusGroupId().toString());
+ seekQueueToRegionProgress(queue, regionProgress, true);
+ }
+ }
+
+ private void seekQueueToRegionProgress(
+ final ConsensusPrefetchingQueue queue,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return;
+ }
+ if (seekAfter) {
+ queue.seekAfterRegionProgress(regionProgress);
+ } else {
+ queue.seekToRegionProgress(regionProgress);
+ }
+ }
+
+ //////////////////////////// prefetching ////////////////////////////
+
+ @Override
+ public boolean executePrefetch(final String topicName) {
+ // Consensus prefetch is fully driven by queue-local wakeup sources and the dedicated delayed
+ // scheduler. This interface remains only to satisfy the shared broker contract used by
+ // pipe-based subscription.
+ return false;
+ }
+
+ @Override
+ public int getEventCount(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues)) {
+ return 0;
+ }
+ return queues.stream()
+ .filter(queue -> !queue.isClosed())
+ .mapToInt(ConsensusPrefetchingQueue::getPrefetchedEventCount)
+ .sum();
+ }
+
+ @Override
+ public int getQueueCount() {
+ return topicNameToConsensusPrefetchingQueues.size();
+ }
+
+ /**
+ * Returns per-region lag information for all topics managed by this broker. The result maps
+ * "topicName/regionId" to the lag (number of WAL entries behind).
+ */
+ public Map<String, Long> getLagSummary() {
+ final Map<String, Long> lagMap = new ConcurrentHashMap<>();
+ for (final Map.Entry<String, List<ConsensusPrefetchingQueue>> entry :
+ topicNameToConsensusPrefetchingQueues.entrySet()) {
+ for (final ConsensusPrefetchingQueue queue : entry.getValue()) {
+ if (!queue.isClosed()) {
+ lagMap.put(entry.getKey() + "/" + queue.getConsensusGroupId().toString(), queue.getLag());
+ }
+ }
+ }
+ return lagMap;
+ }
+
+ private TopicOwnershipSnapshot refreshAndGetTopicOwnership(
+ final String topicName,
+ final List<ConsensusPrefetchingQueue> queues,
+ final String consumerId) {
+ final ConcurrentHashMap<String, Long> consumerTimestamps =
+ topicConsumerLastPollMs.computeIfAbsent(topicName, ignored -> new ConcurrentHashMap<>());
+ consumerTimestamps.put(consumerId, System.currentTimeMillis());
+ evictInactiveConsumers(consumerTimestamps);
+ final List<String> sortedConsumers = new ArrayList<>(consumerTimestamps.keySet());
+ Collections.sort(sortedConsumers);
+
+ final List<String> activeRegionIds =
+ queues.stream()
+ .filter(q -> !q.isClosed())
+ .map(q -> q.getConsensusGroupId().toString())
+ .sorted()
+ .collect(Collectors.toList());
+
+ final TopicOwnershipSnapshot existingSnapshot = topicOwnershipSnapshots.get(topicName);
+ if (Objects.nonNull(existingSnapshot)
+ && existingSnapshot.hasSameConsumers(sortedConsumers)
+ && existingSnapshot.hasSameRegions(activeRegionIds)) {
+ return existingSnapshot;
+ }
+
+ final TopicOwnershipSnapshot refreshedSnapshot =
+ TopicOwnershipSnapshot.create(sortedConsumers, activeRegionIds);
+ topicOwnershipSnapshots.put(topicName, refreshedSnapshot);
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: refreshed ownership for topic [{}], consumers={}, regions={}, generation={}",
+ brokerId,
+ topicName,
+ sortedConsumers,
+ activeRegionIds,
+ refreshedSnapshot.getGeneration());
+ return refreshedSnapshot;
+ }
+
+ private List<ConsensusPrefetchingQueue> getAssignedQueues(
+ final List<ConsensusPrefetchingQueue> queues,
+ final String consumerId,
+ final TopicOwnershipSnapshot ownershipSnapshot) {
+ if (Objects.isNull(ownershipSnapshot) || ownershipSnapshot.isEmpty()) {
+ return Collections.emptyList();
+ }
+ final List<ConsensusPrefetchingQueue> assignedQueues = new ArrayList<>();
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ if (consumerId.equals(
+ ownershipSnapshot.getOwnerConsumerId(queue.getConsensusGroupId().toString()))) {
+ assignedQueues.add(queue);
+ }
+ }
+ return assignedQueues;
+ }
+
+ private List<ConsensusPrefetchingQueue> buildPollOrderForAssignedQueues(
+ final List<ConsensusPrefetchingQueue> assignedQueues, final String topicName) {
+ if (assignedQueues.size() <= 1) {
+ return assignedQueues;
+ }
+ final List<ConsensusPrefetchingQueue> pollQueues = new ArrayList<>(assignedQueues);
+ if (SubscriptionConfig.getInstance().isSubscriptionConsensusLagBasedPriority()) {
+ pollQueues.sort(
+ Comparator.comparingLong(ConsensusPrefetchingQueue::getLag)
+ .reversed()
+ .thenComparing(q -> q.getConsensusGroupId().toString()));
+ return pollQueues;
+ }
+
+ final int startOffset = Math.floorMod(pollRoundRobinIndex.getAndIncrement(), pollQueues.size());
+ final List<ConsensusPrefetchingQueue> orderedQueues = new ArrayList<>(pollQueues.size());
+ for (int i = 0; i < pollQueues.size(); i++) {
+ orderedQueues.add(pollQueues.get((startOffset + i) % pollQueues.size()));
+ }
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: stable ownership poll order for topic [{}], assignedQueueCount={}",
+ brokerId,
+ topicName,
+ orderedQueues.size());
+ return orderedQueues;
+ }
+
+ private ConsensusPrefetchingQueue getAssignedQueueForConsumer(
+ final List<ConsensusPrefetchingQueue> queues,
+ final String topicName,
+ final String consumerId,
+ final String regionId,
+ final String action) {
+ final TopicOwnershipSnapshot ownershipSnapshot =
+ refreshAndGetTopicOwnership(topicName, queues, consumerId);
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ if (!regionId.isEmpty() && !regionId.equals(queue.getConsensusGroupId().toString())) {
+ continue;
+ }
+ if (consumerId.equals(
+ ownershipSnapshot.getOwnerConsumerId(queue.getConsensusGroupId().toString()))) {
+ return queue;
+ }
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: consumer [{}] skipped {} on topic [{}], region [{}] is currently owned by [{}]",
+ brokerId,
+ consumerId,
+ action,
+ topicName,
+ queue.getConsensusGroupId(),
+ ownershipSnapshot.getOwnerConsumerId(queue.getConsensusGroupId().toString()));
+ return null;
+ }
+ return null;
+ }
+
+ /** Evicts consumers that have not polled within the configured eviction timeout. */
+ private void evictInactiveConsumers(final ConcurrentHashMap<String, Long> consumerTimestamps) {
+ final long now = System.currentTimeMillis();
+ final long timeout =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusConsumerEvictionTimeoutMs();
+ consumerTimestamps.entrySet().removeIf(entry -> (now - entry.getValue()) > timeout);
+ }
+
+ //////////////////////////// queue management ////////////////////////////
+
+ public void bindConsensusPrefetchingQueue(
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ // Get or create the list of queues for this topic
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.computeIfAbsent(
+ topicName, k -> new CopyOnWriteArrayList<>());
+
+ // Check for duplicate region binding
+ for (final ConsensusPrefetchingQueue existing : queues) {
+ if (consensusGroupId.equals(existing.getConsensusGroupId()) && !existing.isClosed()) {
+ LOGGER.info(
+ "Subscription: consensus prefetching queue for topic [{}], region [{}] "
+ + "in consumer group [{}] already exists, skipping",
+ topicName,
+ consensusGroupId,
+ brokerId);
+ return;
+ }
+ }
+
+ // Create the per-region consensus queue for this topic.
+ final ConsensusPrefetchingQueue consensusQueue =
+ new ConsensusPrefetchingQueue(
+ brokerId,
+ topicName,
+ orderMode,
+ consensusGroupId,
+ serverImpl,
+ retentionPolicy,
+ converter,
+ commitManager,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+ queues.add(consensusQueue);
+ LOGGER.info(
+ "Subscription: create consensus prefetching queue bound to topic [{}] for consumer group [{}], "
+ + "consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "tailStartSearchIndex={}, initialRuntimeVersion={}, initialActive={}, totalRegionQueues={}",
+ topicName,
+ brokerId,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive,
+ queues.size());
+ }
+
+ public void refreshConsensusQueueOrderMode(final String topicName, final String orderMode) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return;
+ }
+
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ queue.setOrderMode(orderMode);
+ }
+ }
+
+ public void unbindConsensusPrefetchingQueue(final String topicName) {
+ closeAndRemoveConsensusPrefetchingQueues(topicName, true);
+ }
+
+ public int unbindByRegion(final ConsensusGroupId regionId) {
+ int closedCount = 0;
+ for (final Map.Entry<String, List<ConsensusPrefetchingQueue>> entry :
+ topicNameToConsensusPrefetchingQueues.entrySet()) {
+ final List<ConsensusPrefetchingQueue> queues = entry.getValue();
+ final int beforeSize = queues.size();
+ queues.removeIf(
+ q -> {
+ if (!regionId.equals(q.getConsensusGroupId())) {
+ return false;
+ }
+ q.close();
+ LOGGER.info(
+ "Subscription: closed consensus prefetching queue for topic [{}] region [{}] "
+ + "in consumer group [{}] due to region removal",
+ entry.getKey(),
+ regionId,
+ brokerId);
+ return true;
+ });
+ closedCount += beforeSize - queues.size();
+ if (queues.isEmpty()) {
+ topicNameToConsensusPrefetchingQueues.remove(entry.getKey(), queues);
+ topicConsumerLastPollMs.remove(entry.getKey());
+ topicOwnershipSnapshots.remove(entry.getKey());
+ } else {
+ topicOwnershipSnapshots.remove(entry.getKey());
+ }
+ }
+ return closedCount;
+ }
+
+ /**
+ * Activates or deactivates all queues bound to {@code regionId}. Called on leader migration:
+ * {@code false} on old leader, {@code true} on new leader. Inactive queues skip prefetching and
+ * return null on poll, ensuring only the preferred writer serves subscription data.
+ */
+ public void setActiveForRegion(final ConsensusGroupId regionId, final boolean active) {
+ for (final List<ConsensusPrefetchingQueue> queues :
+ topicNameToConsensusPrefetchingQueues.values()) {
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (regionId.equals(q.getConsensusGroupId())) {
+ q.setActive(active);
+ }
+ }
+ }
+ }
+
+ public void setActiveWritersForRegion(
+ final ConsensusGroupId regionId, final Set<Integer> activeWriterNodeIds) {
+ final Set<Integer> normalizedActiveWriterNodeIds =
+ Collections.unmodifiableSet(new LinkedHashSet<>(activeWriterNodeIds));
+ for (final List<ConsensusPrefetchingQueue> queues :
+ topicNameToConsensusPrefetchingQueues.values()) {
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (regionId.equals(q.getConsensusGroupId())) {
+ q.setActiveWriterNodeIds(normalizedActiveWriterNodeIds);
+ }
+ }
+ }
+ }
+
+ public void applyRuntimeStateForRegion(
+ final ConsensusGroupId regionId, final ConsensusRegionRuntimeState runtimeState) {
+ for (final List<ConsensusPrefetchingQueue> queues :
+ topicNameToConsensusPrefetchingQueues.values()) {
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (regionId.equals(q.getConsensusGroupId())) {
+ q.applyRuntimeState(runtimeState);
+ }
+ }
+ }
+ }
+
+ public void abortPendingSeeksForRuntimeStop() {
+ for (final List<ConsensusPrefetchingQueue> queues :
+ topicNameToConsensusPrefetchingQueues.values()) {
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!q.isClosed()) {
+ q.abortPendingSeekForRuntimeStop();
+ }
+ }
+ }
+ }
+
+ @Override
+ public void removeQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.nonNull(queues) && !queues.isEmpty()) {
+ LOGGER.info(
+ "Subscription: consensus prefetching queue(s) bound to topic [{}] for consumer group [{}] still exist, unbind before closing",
+ topicName,
+ brokerId);
+ }
+ closeAndRemoveConsensusPrefetchingQueues(topicName, false);
+ }
Review Comment:
Added a broker-level queue lifecycle lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3346562381
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertTabletNode;
+
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/** Converts IoTConsensus WAL log entries (InsertNode) to Tablet format for subscription. */
+public class ConsensusLogToTabletConverter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusLogToTabletConverter.class);
+
+ private final TreePattern treePattern;
+ private final TablePattern tablePattern;
+ private final Pattern tableColumnPattern;
+
+ /**
+ * The actual database name of the DataRegion this converter processes (table-model format without
+ * "root." prefix). Null for tree-model topics.
+ */
+ private final String databaseName;
+
+ public ConsensusLogToTabletConverter(
+ final TreePattern treePattern,
+ final TablePattern tablePattern,
+ final Pattern tableColumnPattern,
+ final String databaseName) {
+ this.treePattern = treePattern;
+ this.tablePattern = tablePattern;
+ this.tableColumnPattern = tableColumnPattern;
+ this.databaseName = databaseName;
+ }
+
+ public String getDatabaseName() {
+ return databaseName;
+ }
+
+ static String safeDeviceIdForLog(final InsertNode node) {
+ try {
+ final Object deviceId = node.getDeviceID();
+ return deviceId != null ? deviceId.toString() : "null";
+ } catch (final Exception e) {
+ return "N/A(" + node.getType() + ")";
+ }
+ }
+
+ public List<Tablet> convert(final InsertNode insertNode) {
+ if (Objects.isNull(insertNode)) {
+ return Collections.emptyList();
+ }
+
+ final PlanNodeType nodeType = insertNode.getType();
+ if (nodeType == null) {
+ LOGGER.warn("InsertNode type is null, skipping conversion");
+ return Collections.emptyList();
+ }
+
+ LOGGER.debug(
+ "ConsensusLogToTabletConverter: converting InsertNode type={}, deviceId={}",
+ nodeType,
+ safeDeviceIdForLog(insertNode));
+
+ switch (nodeType) {
+ case INSERT_ROW:
+ return convertInsertRowNode((InsertRowNode) insertNode);
+ case INSERT_TABLET:
+ return convertInsertTabletNode((InsertTabletNode) insertNode);
+ case INSERT_ROWS:
+ return convertInsertRowsNode((InsertRowsNode) insertNode);
+ case INSERT_ROWS_OF_ONE_DEVICE:
+ return convertInsertRowsOfOneDeviceNode((InsertRowsOfOneDeviceNode) insertNode);
+ case INSERT_MULTI_TABLET:
+ return convertInsertMultiTabletsNode((InsertMultiTabletsNode) insertNode);
+ case RELATIONAL_INSERT_ROW:
+ return convertRelationalInsertRowNode((RelationalInsertRowNode) insertNode);
+ case RELATIONAL_INSERT_TABLET:
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) insertNode);
+ case RELATIONAL_INSERT_ROWS:
+ return convertRelationalInsertRowsNode((RelationalInsertRowsNode) insertNode);
+ default:
+ LOGGER.debug("Unsupported InsertNode type for subscription: {}", nodeType);
+ return Collections.emptyList();
+ }
+ }
+
+ // ======================== Tree Model Conversion ========================
+
+ private List<Tablet> convertInsertRowNode(final InsertRowNode node) {
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final long time = node.getTime();
+
+ // Determine which columns match the pattern
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final Object[] values = node.getValues();
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Build Tablet with matched columns
+ final int columnCount = matchedColumnIndices.size();
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ final Tablet tablet = new Tablet(deviceId.toString(), schemas, 1 /* maxRowNumber */);
+ tablet.addTimestamp(0, time);
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = matchedColumnIndices.get(i);
+ final Object value = values[originalColIdx];
+ if (value == null) {
+ if (tablet.getBitMaps() == null) {
+ tablet.initBitMaps();
+ }
+ tablet.getBitMaps()[i].mark(0);
+ } else {
+ addValueToTablet(tablet, 0, i, dataTypes[originalColIdx], value);
+ }
+ }
+ tablet.setRowSize(1);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
+ if (node instanceof RelationalInsertTabletNode) {
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) node);
+ }
+
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final long[] times = node.getTimes();
+ final Object[] columns = node.getColumns();
+ final BitMap[] bitMaps = node.getBitMaps();
+ final int rowCount = node.getRowCount();
+
+ // Column filtering
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final int columnCount = matchedColumnIndices.size();
+ final boolean allColumnsMatch = (columnCount == measurements.length);
+
+ // Build schemas (always needed)
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ // Build column arrays and bitmaps using bulk copy
+ final long[] newTimes = Arrays.copyOf(times, rowCount);
+ final Object[] newColumns = new Object[columnCount];
+ final BitMap[] newBitMaps = new BitMap[columnCount];
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = allColumnsMatch ? i : matchedColumnIndices.get(i);
+ newColumns[i] = copyColumnArray(dataTypes[originalColIdx], columns[originalColIdx], rowCount);
+ if (bitMaps != null && bitMaps[originalColIdx] != null) {
+ newBitMaps[i] = new BitMap(rowCount);
+ BitMap.copyOfRange(bitMaps[originalColIdx], 0, newBitMaps[i], 0, rowCount);
+ }
+ }
Review Comment:
Done in 5ea428025e. The complete-tablet path now reuses the original time/column/bitmap arrays on the server-side read path; copies are kept only when column filtering changes the tablet shape. A comment was added to document that tradeoff.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
Review Comment:
Done in 5ea428025e. Renamed PipeSubscribeHeartbeatReq to SubscriptionHeartbeatReq and updated the call sites/log wording accordingly.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) throws IOException {
+ final PipeSubscribeHeartbeatReq req = toTPipeSubscribeReq();
+ req.processorBufferedCommitContexts =
+ Objects.nonNull(processorBufferedCommitContexts)
+ ? processorBufferedCommitContexts
+ : new ArrayList<>();
Review Comment:
Done in 5ea428025e. The default buffered commit context list now uses Collections.emptyList().
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertTabletNode;
+
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/** Converts IoTConsensus WAL log entries (InsertNode) to Tablet format for subscription. */
+public class ConsensusLogToTabletConverter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusLogToTabletConverter.class);
+
+ private final TreePattern treePattern;
+ private final TablePattern tablePattern;
+ private final Pattern tableColumnPattern;
+
+ /**
+ * The actual database name of the DataRegion this converter processes (table-model format without
+ * "root." prefix). Null for tree-model topics.
+ */
+ private final String databaseName;
+
+ public ConsensusLogToTabletConverter(
+ final TreePattern treePattern,
+ final TablePattern tablePattern,
+ final Pattern tableColumnPattern,
+ final String databaseName) {
+ this.treePattern = treePattern;
+ this.tablePattern = tablePattern;
+ this.tableColumnPattern = tableColumnPattern;
+ this.databaseName = databaseName;
+ }
+
+ public String getDatabaseName() {
+ return databaseName;
+ }
+
+ static String safeDeviceIdForLog(final InsertNode node) {
+ try {
+ final Object deviceId = node.getDeviceID();
+ return deviceId != null ? deviceId.toString() : "null";
+ } catch (final Exception e) {
+ return "N/A(" + node.getType() + ")";
+ }
+ }
+
+ public List<Tablet> convert(final InsertNode insertNode) {
+ if (Objects.isNull(insertNode)) {
+ return Collections.emptyList();
+ }
+
+ final PlanNodeType nodeType = insertNode.getType();
+ if (nodeType == null) {
+ LOGGER.warn("InsertNode type is null, skipping conversion");
+ return Collections.emptyList();
+ }
+
+ LOGGER.debug(
+ "ConsensusLogToTabletConverter: converting InsertNode type={}, deviceId={}",
+ nodeType,
+ safeDeviceIdForLog(insertNode));
+
+ switch (nodeType) {
+ case INSERT_ROW:
+ return convertInsertRowNode((InsertRowNode) insertNode);
+ case INSERT_TABLET:
+ return convertInsertTabletNode((InsertTabletNode) insertNode);
+ case INSERT_ROWS:
+ return convertInsertRowsNode((InsertRowsNode) insertNode);
+ case INSERT_ROWS_OF_ONE_DEVICE:
+ return convertInsertRowsOfOneDeviceNode((InsertRowsOfOneDeviceNode) insertNode);
+ case INSERT_MULTI_TABLET:
+ return convertInsertMultiTabletsNode((InsertMultiTabletsNode) insertNode);
+ case RELATIONAL_INSERT_ROW:
+ return convertRelationalInsertRowNode((RelationalInsertRowNode) insertNode);
+ case RELATIONAL_INSERT_TABLET:
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) insertNode);
+ case RELATIONAL_INSERT_ROWS:
+ return convertRelationalInsertRowsNode((RelationalInsertRowsNode) insertNode);
+ default:
+ LOGGER.debug("Unsupported InsertNode type for subscription: {}", nodeType);
+ return Collections.emptyList();
+ }
+ }
+
+ // ======================== Tree Model Conversion ========================
+
+ private List<Tablet> convertInsertRowNode(final InsertRowNode node) {
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final long time = node.getTime();
+
+ // Determine which columns match the pattern
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final Object[] values = node.getValues();
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Build Tablet with matched columns
+ final int columnCount = matchedColumnIndices.size();
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ final Tablet tablet = new Tablet(deviceId.toString(), schemas, 1 /* maxRowNumber */);
+ tablet.addTimestamp(0, time);
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = matchedColumnIndices.get(i);
+ final Object value = values[originalColIdx];
+ if (value == null) {
+ if (tablet.getBitMaps() == null) {
+ tablet.initBitMaps();
+ }
+ tablet.getBitMaps()[i].mark(0);
+ } else {
+ addValueToTablet(tablet, 0, i, dataTypes[originalColIdx], value);
+ }
+ }
+ tablet.setRowSize(1);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
+ if (node instanceof RelationalInsertTabletNode) {
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) node);
+ }
+
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final long[] times = node.getTimes();
+ final Object[] columns = node.getColumns();
+ final BitMap[] bitMaps = node.getBitMaps();
+ final int rowCount = node.getRowCount();
+
+ // Column filtering
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final int columnCount = matchedColumnIndices.size();
+ final boolean allColumnsMatch = (columnCount == measurements.length);
+
+ // Build schemas (always needed)
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ // Build column arrays and bitmaps using bulk copy
+ final long[] newTimes = Arrays.copyOf(times, rowCount);
+ final Object[] newColumns = new Object[columnCount];
+ final BitMap[] newBitMaps = new BitMap[columnCount];
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = allColumnsMatch ? i : matchedColumnIndices.get(i);
+ newColumns[i] = copyColumnArray(dataTypes[originalColIdx], columns[originalColIdx], rowCount);
+ if (bitMaps != null && bitMaps[originalColIdx] != null) {
+ newBitMaps[i] = new BitMap(rowCount);
+ BitMap.copyOfRange(bitMaps[originalColIdx], 0, newBitMaps[i], 0, rowCount);
+ }
+ }
Review Comment:
Done in 5ea428025e. The complete-tablet path now reuses the original time/column/bitmap arrays on the server-side read path; copies are kept only when column filtering changes the tablet shape. A comment was added to document that tradeoff.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
Review Comment:
Done in 5ea428025e. Renamed PipeSubscribeHeartbeatReq to SubscriptionHeartbeatReq and updated the call sites/log wording accordingly.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) throws IOException {
+ final PipeSubscribeHeartbeatReq req = toTPipeSubscribeReq();
+ req.processorBufferedCommitContexts =
+ Objects.nonNull(processorBufferedCommitContexts)
+ ? processorBufferedCommitContexts
+ : new ArrayList<>();
Review Comment:
Done in 5ea428025e. The default buffered commit context list now uses Collections.emptyList().
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
Review Comment:
Done in 5ea428025e. Renamed PipeSubscribeHeartbeatReq to SubscriptionHeartbeatReq and updated the call sites/log wording accordingly.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertTabletNode;
+
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/** Converts IoTConsensus WAL log entries (InsertNode) to Tablet format for subscription. */
+public class ConsensusLogToTabletConverter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusLogToTabletConverter.class);
+
+ private final TreePattern treePattern;
+ private final TablePattern tablePattern;
+ private final Pattern tableColumnPattern;
+
+ /**
+ * The actual database name of the DataRegion this converter processes (table-model format without
+ * "root." prefix). Null for tree-model topics.
+ */
+ private final String databaseName;
+
+ public ConsensusLogToTabletConverter(
+ final TreePattern treePattern,
+ final TablePattern tablePattern,
+ final Pattern tableColumnPattern,
+ final String databaseName) {
+ this.treePattern = treePattern;
+ this.tablePattern = tablePattern;
+ this.tableColumnPattern = tableColumnPattern;
+ this.databaseName = databaseName;
+ }
+
+ public String getDatabaseName() {
+ return databaseName;
+ }
+
+ static String safeDeviceIdForLog(final InsertNode node) {
+ try {
+ final Object deviceId = node.getDeviceID();
+ return deviceId != null ? deviceId.toString() : "null";
+ } catch (final Exception e) {
+ return "N/A(" + node.getType() + ")";
+ }
+ }
+
+ public List<Tablet> convert(final InsertNode insertNode) {
+ if (Objects.isNull(insertNode)) {
+ return Collections.emptyList();
+ }
+
+ final PlanNodeType nodeType = insertNode.getType();
+ if (nodeType == null) {
+ LOGGER.warn("InsertNode type is null, skipping conversion");
+ return Collections.emptyList();
+ }
+
+ LOGGER.debug(
+ "ConsensusLogToTabletConverter: converting InsertNode type={}, deviceId={}",
+ nodeType,
+ safeDeviceIdForLog(insertNode));
+
+ switch (nodeType) {
+ case INSERT_ROW:
+ return convertInsertRowNode((InsertRowNode) insertNode);
+ case INSERT_TABLET:
+ return convertInsertTabletNode((InsertTabletNode) insertNode);
+ case INSERT_ROWS:
+ return convertInsertRowsNode((InsertRowsNode) insertNode);
+ case INSERT_ROWS_OF_ONE_DEVICE:
+ return convertInsertRowsOfOneDeviceNode((InsertRowsOfOneDeviceNode) insertNode);
+ case INSERT_MULTI_TABLET:
+ return convertInsertMultiTabletsNode((InsertMultiTabletsNode) insertNode);
+ case RELATIONAL_INSERT_ROW:
+ return convertRelationalInsertRowNode((RelationalInsertRowNode) insertNode);
+ case RELATIONAL_INSERT_TABLET:
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) insertNode);
+ case RELATIONAL_INSERT_ROWS:
+ return convertRelationalInsertRowsNode((RelationalInsertRowsNode) insertNode);
+ default:
+ LOGGER.debug("Unsupported InsertNode type for subscription: {}", nodeType);
+ return Collections.emptyList();
+ }
+ }
+
+ // ======================== Tree Model Conversion ========================
+
+ private List<Tablet> convertInsertRowNode(final InsertRowNode node) {
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final long time = node.getTime();
+
+ // Determine which columns match the pattern
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final Object[] values = node.getValues();
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Build Tablet with matched columns
+ final int columnCount = matchedColumnIndices.size();
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ final Tablet tablet = new Tablet(deviceId.toString(), schemas, 1 /* maxRowNumber */);
+ tablet.addTimestamp(0, time);
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = matchedColumnIndices.get(i);
+ final Object value = values[originalColIdx];
+ if (value == null) {
+ if (tablet.getBitMaps() == null) {
+ tablet.initBitMaps();
+ }
+ tablet.getBitMaps()[i].mark(0);
+ } else {
+ addValueToTablet(tablet, 0, i, dataTypes[originalColIdx], value);
+ }
+ }
+ tablet.setRowSize(1);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
+ if (node instanceof RelationalInsertTabletNode) {
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) node);
+ }
+
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final long[] times = node.getTimes();
+ final Object[] columns = node.getColumns();
+ final BitMap[] bitMaps = node.getBitMaps();
+ final int rowCount = node.getRowCount();
+
+ // Column filtering
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final int columnCount = matchedColumnIndices.size();
+ final boolean allColumnsMatch = (columnCount == measurements.length);
+
+ // Build schemas (always needed)
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ // Build column arrays and bitmaps using bulk copy
+ final long[] newTimes = Arrays.copyOf(times, rowCount);
+ final Object[] newColumns = new Object[columnCount];
+ final BitMap[] newBitMaps = new BitMap[columnCount];
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = allColumnsMatch ? i : matchedColumnIndices.get(i);
+ newColumns[i] = copyColumnArray(dataTypes[originalColIdx], columns[originalColIdx], rowCount);
+ if (bitMaps != null && bitMaps[originalColIdx] != null) {
+ newBitMaps[i] = new BitMap(rowCount);
+ BitMap.copyOfRange(bitMaps[originalColIdx], 0, newBitMaps[i], 0, rowCount);
+ }
+ }
Review Comment:
Done in 5ea428025e. The complete-tablet path now reuses the original time/column/bitmap arrays on the server-side read path; copies are kept only when column filtering changes the tablet shape. A comment was added to document that tradeoff.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) throws IOException {
+ final PipeSubscribeHeartbeatReq req = toTPipeSubscribeReq();
+ req.processorBufferedCommitContexts =
+ Objects.nonNull(processorBufferedCommitContexts)
+ ? processorBufferedCommitContexts
+ : new ArrayList<>();
Review Comment:
Done in 5ea428025e. The default buffered commit context list now uses Collections.emptyList().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#issuecomment-4619253766
CI note: the two failed `Simple (17)` jobs are failing during `Set up job` before checkout/tests. After rerun they still stop while downloading `actions/checkout@v5` on self-hosted runners (`iotdb-1` / `iotdb-5`) with `tar: command not found`, so this looks like runner environment infrastructure rather than a code failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3246258630
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/receiver/SubscriptionReceiverV1.java:
##########
@@ -669,22 +700,90 @@ private TPipeSubscribeResp handlePipeSubscribeCommitInternal(final PipeSubscribe
if (Objects.equals(successfulCommitContexts.size(), commitContexts.size())) {
LOGGER.info(
- "Subscription: consumer {} commit (nack: {}) successfully, commit contexts: {}",
+ "Subscription: consumer {} commit (nack: {}) successfully, summary: {}",
consumerConfig,
nack,
- commitContexts);
+ summarizeCommitContexts(commitContexts));
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug(
+ "Subscription: consumer {} commit (nack: {}) full commit contexts: {}",
+ consumerConfig,
+ nack,
+ commitContexts);
+ }
} else {
LOGGER.warn(
- "Subscription: consumer {} commit (nack: {}) partially successful, commit contexts: {}, successful commit contexts: {}",
+ "Subscription: consumer {} commit (nack: {}) partially successful, requested summary: {}, successful summary: {}",
consumerConfig,
nack,
- commitContexts,
- successfulCommitContexts);
+ summarizeCommitContexts(commitContexts),
+ summarizeCommitContexts(successfulCommitContexts));
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug(
+ "Subscription: consumer {} commit (nack: {}) full requested commit contexts: {}, full successful commit contexts: {}",
+ consumerConfig,
+ nack,
+ commitContexts,
+ successfulCommitContexts);
+ }
}
return PipeSubscribeCommitResp.toTPipeSubscribeResp(RpcUtils.SUCCESS_STATUS);
Review Comment:
Fixed. The commit response now separates accepted commit contexts from the server-side durable committed progress. The client only clears pending ACKs using accepted contexts, and only advances local committed positions using the committed progress returned by the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2922136478
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/receiver/SubscriptionReceiverV1.java:
##########
@@ -662,6 +666,45 @@ private TPipeSubscribeResp handlePipeSubscribeCloseInternal(final PipeSubscribeC
return PipeSubscribeCloseResp.toTPipeSubscribeResp(RpcUtils.SUCCESS_STATUS);
}
+ private TPipeSubscribeResp handlePipeSubscribeSeek(final PipeSubscribeSeekReq req) {
+ try {
+ return handlePipeSubscribeSeekInternal(req);
+ } catch (final Exception e) {
+ LOGGER.warn("Exception occurred when seeking with request {}", req, e);
+ final String exceptionMessage =
+ String.format(
+ "Subscription: something unexpected happened when seeking with request %s: %s",
+ req, e);
+ return PipeSubscribeSeekResp.toTPipeSubscribeResp(
+ RpcUtils.getStatus(TSStatusCode.SUBSCRIPTION_SEEK_ERROR, exceptionMessage));
+ }
+ }
+
+ private TPipeSubscribeResp handlePipeSubscribeSeekInternal(final PipeSubscribeSeekReq req) {
+ // check consumer config thread local
+ final ConsumerConfig consumerConfig = consumerConfigThreadLocal.get();
+ if (Objects.isNull(consumerConfig)) {
+ LOGGER.warn(
+ "Subscription: missing consumer config when handling PipeSubscribeSeekReq: {}", req);
+ return SUBSCRIPTION_MISSING_CUSTOMER_RESP;
Review Comment:
MISSING_CONSUMER_RESP?
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java:
##########
@@ -389,6 +389,13 @@ public class CommonConfig {
private long subscriptionMetaSyncerInitialSyncDelayMinutes = 3;
private long subscriptionMetaSyncerSyncIntervalMinutes = 3;
+ private int subscriptionConsensusBatchMaxDelayInMs = 50;
+ private long subscriptionConsensusBatchMaxSizeInBytes = 8 * MB;
+ private int subscriptionConsensusBatchMaxTabletCount = 64;
+ private int subscriptionConsensusBatchMaxWalEntries = 128;
+
+ private long subscriptionConsensusWalRetentionSizeInBytes = 512 * MB;
Review Comment:
May consider "throttleThreshold" as default value or directly use that?
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java:
##########
@@ -872,17 +955,41 @@ void checkAndUpdateIndex() {
}
/**
- * If there is only one replica, set it to Long.MAX_VALUE. If there are multiple replicas, get the
- * latest SafelyDeletedSearchIndex again. This enables wal to be deleted in a timely manner.
+ * Computes and updates the safe-to-delete WAL search index based on replication progress and
+ * subscription WAL retention policy. When no subscriptions exist, WAL is cleaned normally.
*/
- void checkAndUpdateSafeDeletedSearchIndex() {
+ public void checkAndUpdateSafeDeletedSearchIndex() {
if (configuration.isEmpty()) {
logger.error(
"Configuration is empty, which is unexpected. Safe deleted search index won't be updated this time.");
- } else if (configuration.size() == 1) {
+ return;
+ }
+
+ final boolean hasSubscriptions = !subscriptionQueues.isEmpty();
+ final long retentionSizeLimit =
+ config.getReplication().getSubscriptionWalRetentionSizeInBytes();
+
+ if (configuration.size() == 1 && !hasSubscriptions) {
+ // Single replica, no subscription consumers => delete all WAL freely
consensusReqReader.setSafelyDeletedSearchIndex(Long.MAX_VALUE);
} else {
- consensusReqReader.setSafelyDeletedSearchIndex(getMinFlushedSyncIndex());
+ final long replicationIndex =
+ configuration.size() > 1 ? getMinFlushedSyncIndex() : Long.MAX_VALUE;
+
+ // Subscription WAL retention: if subscriptions exist and retention is configured,
+ // prevent WAL deletion when total WAL size is within the retention limit.
+ long subscriptionRetentionBound = Long.MAX_VALUE;
+ if (hasSubscriptions && retentionSizeLimit > 0) {
+ final long totalWalSize = consensusReqReader.getTotalSize();
+ if (totalWalSize <= retentionSizeLimit) {
+ // WAL size is within retention limit — preserve all WAL for subscribers
+ subscriptionRetentionBound = ConsensusReqReader.DEFAULT_SAFELY_DELETED_SEARCH_INDEX;
Review Comment:
Seemingly a little bit coarse?
1. The wal clean is either "all" or "not", does not have a bound like "only reserve the retention size amount of data"
2. The reserving logic does not consider the commit indexes of subscriptions
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/subscription/config/SubscriptionConfig.java:
##########
@@ -30,7 +30,7 @@ public class SubscriptionConfig {
private static final CommonConfig COMMON_CONFIG = CommonDescriptor.getInstance().getConfig();
public boolean getSubscriptionEnabled() {
- return false;
+ return true; // TODO: make it configurable after subscription is stable
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3246255396
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionPullConsumer.java:
##########
@@ -195,6 +219,35 @@ protected List<SubscriptionMessage> poll(final Set<String> topicNames, final lon
return messages;
}
+ // Apply processor chain if configured
+ List<SubscriptionMessage> processed = messages;
+ if (!processors.isEmpty()) {
+ for (final SubscriptionMessageProcessor processor : processors) {
+ processed = processor.process(processed);
+ }
+ }
+
+ // Update watermark timestamp before stripping watermark events
+ for (final SubscriptionMessage m : processed) {
+ if (m.getMessageType() == SubscriptionMessageType.WATERMARK.getType()) {
+ final long ts = m.getWatermarkTimestamp();
+ if (ts > latestWatermarkTimestamp) {
+ latestWatermarkTimestamp = ts;
+ }
+ }
+ }
+
+ // Strip system messages — they are only for processors, not for users
+ processed.removeIf(
+ m -> {
+ final short type = m.getMessageType();
+ return type == SubscriptionMessageType.WATERMARK.getType();
+ });
+
+ if (processed.isEmpty()) {
+ return processed;
+ }
+
// add to uncommitted messages
Review Comment:
Fixed. Processor-buffered commit contexts are now tracked explicitly and refreshed through heartbeat, so messages held inside processors remain pinned on the server until they are emitted/released. Topic-scoped reset was also added to avoid clearing unrelated buffered state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279185039
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -0,0 +1,1253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.common.rpc.thrift.TDataNodeLocation;
+import org.apache.iotdb.common.rpc.thrift.TEndPoint;
+import org.apache.iotdb.common.rpc.thrift.TRegionReplicaSet;
+import org.apache.iotdb.commons.client.ClientPoolFactory;
+import org.apache.iotdb.commons.client.IClientManager;
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient;
+import org.apache.iotdb.commons.consensus.ConfigRegionId;
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.confignode.rpc.thrift.TGetCommitProgressReq;
+import org.apache.iotdb.confignode.rpc.thrift.TGetCommitProgressResp;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.protocol.client.ConfigNodeClient;
+import org.apache.iotdb.db.protocol.client.ConfigNodeClientManager;
+import org.apache.iotdb.db.protocol.client.ConfigNodeInfo;
+import org.apache.iotdb.db.queryengine.plan.analyze.ClusterPartitionFetcher;
+import org.apache.iotdb.mpp.rpc.thrift.TSyncSubscriptionProgressReq;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Manages committed progress for consensus-based subscriptions.
+ *
+ * <p>State is maintained per {@code (consumerGroup, topic, region)} so each DataRegion can recover
+ * independently.
+ *
+ * <p>Committed progress is represented in per-writer terms via {@link WriterId} and {@link
+ * WriterProgress}. Outstanding deliveries are tracked by writer-local slots, while commit
+ * advancement is computed with ordered progress keys derived from {@code physicalTime}, {@code
+ * writerNodeId}, and {@code localSeq}. {@code searchIndex} is not the committed frontier here; it
+ * only remains an implementation aid for WAL positioning elsewhere.
+ *
+ * <p>Key responsibilities:
+ *
+ * <ul>
+ * <li>Track dispatched but uncommitted mappings per writer
+ * <li>Advance committed progress idempotently and contiguously on ack/commit
+ * <li>Persist, recover, and broadcast committed region progress
+ * </ul>
+ */
+public class ConsensusSubscriptionCommitManager {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(ConsensusSubscriptionCommitManager.class);
+
+ private static final String PROGRESS_FILE_PREFIX = "consensus_subscription_progress_";
+ private static final String PROGRESS_FILE_SUFFIX = ".dat";
+
+ private static final IClientManager<ConfigRegionId, ConfigNodeClient> CONFIG_NODE_CLIENT_MANAGER =
+ ConfigNodeClientManager.getInstance();
+
+ /** Client manager for DataNode-to-DataNode RPC (progress broadcast). */
+ private static final IClientManager<TEndPoint, SyncDataNodeInternalServiceClient>
+ SYNC_DN_CLIENT_MANAGER =
+ new IClientManager.Factory<TEndPoint, SyncDataNodeInternalServiceClient>()
+ .createClientManager(
+ new ClientPoolFactory.SyncDataNodeInternalServiceClientPoolFactory());
+
+ /** Minimum interval (ms) between broadcasts for the same (consumerGroup, topic, region). */
+ private static final long MIN_BROADCAST_INTERVAL_MS = 5000;
+
+ /** Rate-limiting: last broadcast timestamp per key. */
+ private final Map<String, Long> lastBroadcastTime = new ConcurrentHashMap<>();
+
+ /** Single-threaded executor for fire-and-forget broadcasts. */
+ private final ExecutorService broadcastExecutor =
+ Executors.newSingleThreadExecutor(
+ r -> {
+ final Thread t = new Thread(r, "SubscriptionProgressBroadcast");
+ t.setDaemon(true);
+ return t;
+ });
+
+ /** Key: "consumerGroupId##topicName##regionId" -> progress tracking state */
+ private final Map<String, ConsensusSubscriptionCommitState> commitStates =
+ new ConcurrentHashMap<>();
+
+ private final String persistDir;
+
+ private ConsensusSubscriptionCommitManager() {
+ this.persistDir =
+ IoTDBDescriptor.getInstance().getConfig().getSystemDir()
+ + File.separator
+ + "subscription"
+ + File.separator
+ + "consensus_progress";
+ final File dir = new File(persistDir);
+ if (!dir.exists()) {
+ dir.mkdirs();
+ }
+ }
+
+ /**
+ * Gets or creates the commit state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID
+ * @return the commit state
+ */
+ public ConsensusSubscriptionCommitState getOrCreateState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final String regionIdString = regionId.toString();
+ return commitStates.computeIfAbsent(
+ key,
+ k -> {
+ // Try to recover from persisted local state
+ final ConsensusSubscriptionCommitState recovered = tryRecover(key, regionIdString);
+ if (recovered != null) {
+ return recovered;
+ }
+ final ConsensusSubscriptionCommitState recoveredFromConfigNode =
+ queryCommitProgressStateFromConfigNode(consumerGroupId, topicName, regionId);
+ if (Objects.nonNull(recoveredFromConfigNode)) {
+ return recoveredFromConfigNode;
+ }
+ return new ConsensusSubscriptionCommitState(
+ regionIdString, new SubscriptionConsensusProgress());
+ });
+ }
+
+ public boolean hasPersistedState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ return getProgressFile(generateKey(consumerGroupId, topicName, regionId)).exists();
+ }
+
+ public void recordMapping(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final ConsensusSubscriptionCommitState state =
+ getOrCreateState(consumerGroupId, topicName, regionId);
+ state.recordMapping(writerId, writerProgress);
+ }
+
+ public boolean commit(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot commit for unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionId,
+ writerId,
+ writerProgress);
+ return false;
+ }
+ final CommitOperationResult result = state.commitAndGetResult(writerId, writerProgress);
+ if (result.isHandled()) {
+ // Periodically persist progress
+ persistProgressIfNeeded(key, state);
+ if (result.hasAdvancedWriter()) {
+ maybeBroadcast(
+ key,
+ consumerGroupId,
+ topicName,
+ regionId,
+ result.getAdvancedWriterProgress(),
+ result.getAdvancedWriterId());
+ }
+ }
+ return result.isHandled();
+ }
+
+ public boolean commitWithoutOutstanding(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
Review Comment:
I kept this unbatched for now. The current path already rate-limits persistence and broadcasts, and local persistence can update unchanged-size region payloads in place, so batching would add coordination complexity without a clear benefit yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279194324
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -0,0 +1,1253 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.common.rpc.thrift.TDataNodeLocation;
+import org.apache.iotdb.common.rpc.thrift.TEndPoint;
+import org.apache.iotdb.common.rpc.thrift.TRegionReplicaSet;
+import org.apache.iotdb.commons.client.ClientPoolFactory;
+import org.apache.iotdb.commons.client.IClientManager;
+import org.apache.iotdb.commons.client.exception.ClientManagerException;
+import org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient;
+import org.apache.iotdb.commons.consensus.ConfigRegionId;
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.confignode.rpc.thrift.TGetCommitProgressReq;
+import org.apache.iotdb.confignode.rpc.thrift.TGetCommitProgressResp;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.protocol.client.ConfigNodeClient;
+import org.apache.iotdb.db.protocol.client.ConfigNodeClientManager;
+import org.apache.iotdb.db.protocol.client.ConfigNodeInfo;
+import org.apache.iotdb.db.queryengine.plan.analyze.ClusterPartitionFetcher;
+import org.apache.iotdb.mpp.rpc.thrift.TSyncSubscriptionProgressReq;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/**
+ * Manages committed progress for consensus-based subscriptions.
+ *
+ * <p>State is maintained per {@code (consumerGroup, topic, region)} so each DataRegion can recover
+ * independently.
+ *
+ * <p>Committed progress is represented in per-writer terms via {@link WriterId} and {@link
+ * WriterProgress}. Outstanding deliveries are tracked by writer-local slots, while commit
+ * advancement is computed with ordered progress keys derived from {@code physicalTime}, {@code
+ * writerNodeId}, and {@code localSeq}. {@code searchIndex} is not the committed frontier here; it
+ * only remains an implementation aid for WAL positioning elsewhere.
+ *
+ * <p>Key responsibilities:
+ *
+ * <ul>
+ * <li>Track dispatched but uncommitted mappings per writer
+ * <li>Advance committed progress idempotently and contiguously on ack/commit
+ * <li>Persist, recover, and broadcast committed region progress
+ * </ul>
+ */
+public class ConsensusSubscriptionCommitManager {
+
+ private static final Logger LOGGER =
+ LoggerFactory.getLogger(ConsensusSubscriptionCommitManager.class);
+
+ private static final String PROGRESS_FILE_PREFIX = "consensus_subscription_progress_";
+ private static final String PROGRESS_FILE_SUFFIX = ".dat";
+
+ private static final IClientManager<ConfigRegionId, ConfigNodeClient> CONFIG_NODE_CLIENT_MANAGER =
+ ConfigNodeClientManager.getInstance();
+
+ /** Client manager for DataNode-to-DataNode RPC (progress broadcast). */
+ private static final IClientManager<TEndPoint, SyncDataNodeInternalServiceClient>
+ SYNC_DN_CLIENT_MANAGER =
+ new IClientManager.Factory<TEndPoint, SyncDataNodeInternalServiceClient>()
+ .createClientManager(
+ new ClientPoolFactory.SyncDataNodeInternalServiceClientPoolFactory());
+
+ /** Minimum interval (ms) between broadcasts for the same (consumerGroup, topic, region). */
+ private static final long MIN_BROADCAST_INTERVAL_MS = 5000;
+
+ /** Rate-limiting: last broadcast timestamp per key. */
+ private final Map<String, Long> lastBroadcastTime = new ConcurrentHashMap<>();
+
+ /** Single-threaded executor for fire-and-forget broadcasts. */
+ private final ExecutorService broadcastExecutor =
+ Executors.newSingleThreadExecutor(
+ r -> {
+ final Thread t = new Thread(r, "SubscriptionProgressBroadcast");
+ t.setDaemon(true);
+ return t;
+ });
+
+ /** Key: "consumerGroupId##topicName##regionId" -> progress tracking state */
+ private final Map<String, ConsensusSubscriptionCommitState> commitStates =
+ new ConcurrentHashMap<>();
+
+ private final String persistDir;
+
+ private ConsensusSubscriptionCommitManager() {
+ this.persistDir =
+ IoTDBDescriptor.getInstance().getConfig().getSystemDir()
+ + File.separator
+ + "subscription"
+ + File.separator
+ + "consensus_progress";
+ final File dir = new File(persistDir);
+ if (!dir.exists()) {
+ dir.mkdirs();
+ }
+ }
+
+ /**
+ * Gets or creates the commit state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID
+ * @return the commit state
+ */
+ public ConsensusSubscriptionCommitState getOrCreateState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final String regionIdString = regionId.toString();
+ return commitStates.computeIfAbsent(
+ key,
+ k -> {
+ // Try to recover from persisted local state
+ final ConsensusSubscriptionCommitState recovered = tryRecover(key, regionIdString);
+ if (recovered != null) {
+ return recovered;
+ }
+ final ConsensusSubscriptionCommitState recoveredFromConfigNode =
+ queryCommitProgressStateFromConfigNode(consumerGroupId, topicName, regionId);
+ if (Objects.nonNull(recoveredFromConfigNode)) {
+ return recoveredFromConfigNode;
+ }
+ return new ConsensusSubscriptionCommitState(
+ regionIdString, new SubscriptionConsensusProgress());
+ });
+ }
+
+ public boolean hasPersistedState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ return getProgressFile(generateKey(consumerGroupId, topicName, regionId)).exists();
+ }
+
+ public void recordMapping(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final ConsensusSubscriptionCommitState state =
+ getOrCreateState(consumerGroupId, topicName, regionId);
+ state.recordMapping(writerId, writerProgress);
+ }
+
+ public boolean commit(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot commit for unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionId,
+ writerId,
+ writerProgress);
+ return false;
+ }
+ final CommitOperationResult result = state.commitAndGetResult(writerId, writerProgress);
+ if (result.isHandled()) {
+ // Periodically persist progress
+ persistProgressIfNeeded(key, state);
+ if (result.hasAdvancedWriter()) {
+ maybeBroadcast(
+ key,
+ consumerGroupId,
+ topicName,
+ regionId,
+ result.getAdvancedWriterProgress(),
+ result.getAdvancedWriterId());
+ }
+ }
+ return result.isHandled();
+ }
+
+ public boolean commitWithoutOutstanding(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot direct-commit for unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionId,
+ writerId,
+ writerProgress);
+ return false;
+ }
+ final CommitOperationResult result =
+ state.commitWithoutOutstandingAndGetResult(writerId, writerProgress);
+ if (result.isHandled()) {
+ persistProgressIfNeeded(key, state);
+ if (result.hasAdvancedWriter()) {
+ maybeBroadcast(
+ key,
+ consumerGroupId,
+ topicName,
+ regionId,
+ result.getAdvancedWriterProgress(),
+ result.getAdvancedWriterId());
+ }
+ }
+ return result.isHandled();
+ }
+
+ public long getCommittedPhysicalTime(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedPhysicalTime() : 0L;
+ }
+
+ public long getCommittedLocalSeq(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedLocalSeq() : -1L;
+ }
+
+ public int getCommittedWriterNodeId(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedWriterNodeId() : -1;
+ }
+
+ public WriterId getCommittedWriterId(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedWriterId() : null;
+ }
+
+ public WriterProgress getCommittedWriterProgress(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ return state != null ? state.getCommittedWriterProgress() : null;
+ }
+
+ public RegionProgress getCommittedRegionProgress(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+ return state.getCommittedRegionProgress();
+ }
+
+ /**
+ * Removes state for a specific (consumerGroup, topic, region) triple.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ * @param regionId the consensus group / data region ID
+ */
+ public void removeState(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ commitStates.remove(key);
+ // Clean up persisted file
+ final File file = getProgressFile(key);
+ if (file.exists()) {
+ file.delete();
+ }
+ }
+
+ /**
+ * Removes all states for a given (consumerGroup, topic) pair across all regions. Used during
+ * subscription teardown when the individual regionIds may not be readily available.
+ *
+ * @param consumerGroupId the consumer group ID
+ * @param topicName the topic name
+ */
+ public void removeAllStatesForTopic(final String consumerGroupId, final String topicName) {
+ final String prefix = consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR;
+ final Iterator<Map.Entry<String, ConsensusSubscriptionCommitState>> it =
+ commitStates.entrySet().iterator();
+ while (it.hasNext()) {
+ final Map.Entry<String, ConsensusSubscriptionCommitState> entry = it.next();
+ if (entry.getKey().startsWith(prefix)) {
+ it.remove();
+ final File file = getProgressFile(entry.getKey());
+ if (file.exists()) {
+ file.delete();
+ }
+ }
+ }
+ }
+
+ public void resetState(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final RegionProgress regionProgress) {
+ final String key = generateKey(consumerGroupId, topicName, regionId);
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state == null) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: Cannot reset unknown state, "
+ + "consumerGroupId={}, topicName={}, regionId={}",
+ consumerGroupId,
+ topicName,
+ regionId);
+ return;
+ }
+ state.resetForSeek(regionProgress);
+ persistProgress(key, state);
+ }
+
+ /** Persists all states. Should be called during graceful shutdown. */
+ public void persistAll() {
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ commitStates.entrySet()) {
+ persistProgress(entry.getKey(), entry.getValue());
+ }
+ }
+
+ public Map<String, ByteBuffer> collectAllRegionProgress(final int dataNodeId) {
+ final Map<String, ByteBuffer> result = new ConcurrentHashMap<>();
+ final String suffix = KEY_SEPARATOR + dataNodeId;
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ commitStates.entrySet()) {
+ final RegionProgress regionProgress = entry.getValue().getCommittedRegionProgress();
+ final ByteBuffer serialized = serializeRegionProgress(regionProgress);
+ if (Objects.nonNull(serialized)) {
+ result.put(entry.getKey() + suffix, serialized);
+ }
+ }
+ return result;
+ }
+
+ // ======================== Progress Broadcast (Leader → Follower) ========================
+
+ /**
+ * Broadcasts committed progress to followers if enough time has elapsed since the last broadcast
+ * for this key. The broadcast is async and fire-and-forget.
+ */
+ private void maybeBroadcast(
+ final String key,
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterProgress committedWriterProgress,
+ final WriterId committedWriterId) {
+ if (Objects.isNull(committedWriterId) || Objects.isNull(committedWriterProgress)) {
+ return;
+ }
+ final String broadcastKey = buildBroadcastKey(key, committedWriterId);
+ final long now = System.currentTimeMillis();
+ final Long last = lastBroadcastTime.get(broadcastKey);
+ if (last != null && now - last < MIN_BROADCAST_INTERVAL_MS) {
+ return;
+ }
+ lastBroadcastTime.put(broadcastKey, now);
+ broadcastExecutor.submit(
+ () ->
+ doBroadcast(
+ consumerGroupId, topicName, regionId, committedWriterProgress, committedWriterId));
+ }
+
+ /**
+ * Sends committed progress to all follower replicas of the given region. Uses the partition cache
+ * to discover replica endpoints and skips the local DataNode.
+ */
+ private void doBroadcast(
+ final String consumerGroupId,
+ final String topicName,
+ final ConsensusGroupId regionId,
+ final WriterProgress writerProgress,
+ final WriterId writerId) {
+ final int localDataNodeId = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ try {
+ final List<TRegionReplicaSet> replicaSets =
+ ClusterPartitionFetcher.getInstance()
+ .getRegionReplicaSet(
+ Collections.singletonList(regionId.convertToTConsensusGroupId()));
+ if (replicaSets.isEmpty()) {
+ return;
+ }
+ final String regionIdStr = regionId.toString();
+ final int writerNodeId =
+ Objects.nonNull(writerId) && writerId.getNodeId() >= 0 ? writerId.getNodeId() : -1;
+ final TSyncSubscriptionProgressReq req =
+ new TSyncSubscriptionProgressReq(
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ Objects.nonNull(writerProgress) ? writerProgress.getPhysicalTime() : 0L,
+ writerNodeId,
+ Objects.nonNull(writerProgress) ? writerProgress.getLocalSeq() : -1L);
+
+ for (final TDataNodeLocation location : replicaSets.get(0).getDataNodeLocations()) {
+ if (location.getDataNodeId() == localDataNodeId) {
+ continue; // skip self
+ }
+ final TEndPoint endpoint = location.getInternalEndPoint();
+ try (final SyncDataNodeInternalServiceClient client =
+ SYNC_DN_CLIENT_MANAGER.borrowClient(endpoint)) {
+ client.syncSubscriptionProgress(req);
+ } catch (final ClientManagerException | TException e) {
+ LOGGER.debug(
+ "Failed to broadcast subscription progress to DataNode {} at {}: {}",
+ location.getDataNodeId(),
+ endpoint,
+ e.getMessage());
+ }
+ }
+ } catch (final Exception e) {
+ LOGGER.debug(
+ "Failed to broadcast subscription progress for region {}: {}", regionId, e.getMessage());
+ }
+ }
+
+ /**
+ * Receives a committed progress broadcast from another DataNode (Leader). Updates local state if
+ * the broadcast progress is ahead of the current local progress.
+ */
+ public void receiveProgressBroadcast(
+ final String consumerGroupId,
+ final String topicName,
+ final String regionIdStr,
+ final long physicalTime,
+ final int writerNodeId,
+ final long localSeq) {
+ receiveProgressBroadcast(
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ buildWriterId(regionIdStr, writerNodeId),
+ new WriterProgress(physicalTime, localSeq));
+ }
+
+ public void receiveProgressBroadcast(
+ final String consumerGroupId,
+ final String topicName,
+ final String regionIdStr,
+ final WriterId writerId,
+ final WriterProgress writerProgress) {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ LOGGER.warn(
+ "ConsensusSubscriptionCommitManager: ignore broadcast without writer identity, "
+ + "consumerGroupId={}, topicName={}, regionId={}, writerId={}, writerProgress={}",
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ writerId,
+ writerProgress);
+ return;
+ }
+ final String key = consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionIdStr;
+ final ConsensusSubscriptionCommitState state = commitStates.get(key);
+ if (state != null) {
+ // Update only if broadcast is ahead
+ state.updateFromBroadcast(writerId, writerProgress);
+ persistProgressIfNeeded(key, state);
+ } else {
+ // Create a new state from the broadcast progress
+ final ConsensusSubscriptionCommitState newState =
+ new ConsensusSubscriptionCommitState(
+ regionIdStr,
+ new SubscriptionConsensusProgress(
+ new RegionProgress(Collections.singletonMap(writerId, writerProgress)), 0L));
+ newState.updateFromBroadcast(writerId, writerProgress);
+ commitStates.putIfAbsent(key, newState);
+ persistProgress(key, commitStates.get(key));
+ }
+ LOGGER.debug(
+ "Received subscription progress broadcast: consumerGroupId={}, topicName={}, "
+ + "regionId={}, physicalTime={}, localSeq={}",
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ writerProgress != null ? writerProgress.getPhysicalTime() : 0L,
+ writerProgress != null ? writerProgress.getLocalSeq() : -1L);
+ }
+
+ // ======================== Helper Methods ========================
+
+ // Use a separator that cannot appear in consumerGroupId, topicName, or regionId
+ // to prevent key collisions (e.g., "a_b" + "c" vs "a" + "b_c").
+ private static final String KEY_SEPARATOR = "##";
+
+ private String generateKey(
+ final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
+ return consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionId.toString();
+ }
+
+ private File getProgressFile(final String key) {
+ return new File(persistDir, PROGRESS_FILE_PREFIX + key + PROGRESS_FILE_SUFFIX);
+ }
+
+ private ConsensusSubscriptionCommitState tryRecover(final String key, final String regionIdStr) {
+ final File file = getProgressFile(key);
Review Comment:
Implemented the consumer_group_id##topic_name.meta file. The .index is also implemented, but is memory-only for convenience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279200948
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ProgressWALIterator.java:
##########
@@ -0,0 +1,515 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntryType;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALInfoEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALFileVersion;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.EOFException;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Set;
+
+/**
+ * Writer-based WAL iterator for the new subscription progress model.
+ *
+ * <p>This iterator reads writer-local ordering metadata from WAL footer arrays instead of relying
+ * on the entry body to carry complete subscription ordering information.
+ */
+public class ProgressWALIterator implements Closeable {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ProgressWALIterator.class);
+
+ private static final int SEARCH_INDEX_OFFSET =
+ WALInfoEntry.FIXED_SERIALIZED_SIZE + PlanNodeType.BYTES;
+ private static final long HEADER_ONLY_WAL_FILE_BYTES =
+ Math.max(
+ WALFileVersion.V2.getVersionBytes().length, WALFileVersion.V3.getVersionBytes().length);
+
+ private final File logDirectory;
+ private final long startSearchIndex;
+ private final WALNode liveWalNode;
+ private File[] walFiles;
+ private int currentFileIndex = -1;
+ private ProgressWALReader currentReader;
+ private long currentReaderVersionId = -1L;
+ private boolean currentReaderUsesLiveSnapshot = false;
+ private int consumedEntryCountInCurrentFile = 0;
+ private final Set<Long> skippedBrokenWalVersionIds = new HashSet<>();
+ private IOException lastError;
+ private boolean incompleteScan = false;
+ private String incompleteScanDetail;
+
+ private long pendingSearchIndex = Long.MIN_VALUE;
+ private long pendingLocalSeq = Long.MIN_VALUE;
+ private long pendingPhysicalTime;
+ private int pendingNodeId;
+ private final List<IConsensusRequest> pendingRequests = new ArrayList<>();
+
+ private IndexedConsensusRequest nextReady;
+
+ public ProgressWALIterator(final File logDirectory) {
+ this(logDirectory, Long.MIN_VALUE);
+ }
+
+ public ProgressWALIterator(final File logDirectory, final long startSearchIndex) {
+ this(logDirectory, startSearchIndex, null);
+ }
+
+ public ProgressWALIterator(final WALNode liveWalNode) {
+ this(liveWalNode, Long.MIN_VALUE);
+ }
+
+ public ProgressWALIterator(final WALNode liveWalNode, final long startSearchIndex) {
+ this(liveWalNode.getLogDirectory(), startSearchIndex, liveWalNode);
+ }
+
+ private ProgressWALIterator(
+ final File logDirectory, final long startSearchIndex, final WALNode liveWalNode) {
+ this.logDirectory = logDirectory;
+ this.startSearchIndex = startSearchIndex;
+ this.liveWalNode = liveWalNode;
+ refreshFileList();
+ }
+
+ private void refreshFileList() {
+ final File[] discoveredWalFiles = WALFileUtils.listAllWALFiles(logDirectory);
+ if (discoveredWalFiles == null) {
+ walFiles = new File[0];
+ return;
+ }
+ WALFileUtils.ascSortByVersionId(discoveredWalFiles);
+ final List<File> filteredWalFiles = new ArrayList<>(discoveredWalFiles.length);
+ for (int i = 0; i < discoveredWalFiles.length; i++) {
+ final File walFile = discoveredWalFiles[i];
+ final boolean isLastWalFile = i == discoveredWalFiles.length - 1;
+ if (!isLastWalFile && shouldSkipWalFile(walFile)) {
+ continue;
+ }
+ filteredWalFiles.add(walFile);
+ }
+ walFiles = filteredWalFiles.toArray(new File[0]);
+ }
+
+ private boolean shouldSkipWalFile(final File walFile) {
+ final long versionId = WALFileUtils.parseVersionId(walFile.getName());
+ return skippedBrokenWalVersionIds.contains(versionId) || isHeaderOnlyWalFile(walFile);
+ }
+
+ private boolean isHeaderOnlyWalFile(final File walFile) {
+ return walFile.length() <= HEADER_ONLY_WAL_FILE_BYTES;
+ }
+
+ public void refresh() {
+ final long currentVersionId =
+ (currentFileIndex >= 0 && currentFileIndex < walFiles.length)
+ ? WALFileUtils.parseVersionId(walFiles[currentFileIndex].getName())
+ : -1;
+
+ refreshFileList();
+
+ if (currentVersionId >= 0) {
+ currentFileIndex = -1;
+ for (int i = 0; i < walFiles.length; i++) {
+ if (WALFileUtils.parseVersionId(walFiles[i].getName()) >= currentVersionId) {
+ currentFileIndex = i;
+ break;
+ }
+ }
+ if (currentFileIndex < 0) {
+ currentFileIndex = walFiles.length;
+ }
+ }
+ }
+
+ public boolean hasNext() {
+ if (nextReady != null) {
+ return true;
+ }
+ try {
+ nextReady = advance();
+ if (nextReady != null) {
+ lastError = null;
+ }
+ } catch (IOException e) {
+ lastError = e;
+ LOGGER.warn("ProgressWALIterator: error reading WAL", e);
+ return false;
+ }
+ return nextReady != null;
+ }
+
+ public IndexedConsensusRequest next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+ final IndexedConsensusRequest result = nextReady;
+ nextReady = null;
+ return result;
+ }
+
+ public boolean hasReadError() {
+ return lastError != null;
+ }
+
+ public IOException getLastError() {
+ return lastError;
+ }
+
+ public boolean hasSkippedBrokenWalFiles() {
+ return !skippedBrokenWalVersionIds.isEmpty();
+ }
+
+ public boolean hasIncompleteScan() {
+ return incompleteScan || hasReadError() || hasSkippedBrokenWalFiles();
+ }
+
+ public String getIncompleteScanDetail() {
+ if (incompleteScanDetail != null) {
+ return incompleteScanDetail;
+ }
+ if (lastError != null) {
+ return lastError.getMessage();
+ }
+ if (!skippedBrokenWalVersionIds.isEmpty()) {
+ return "encountered broken retained WAL files during replay scan";
+ }
+ return "replay scan did not complete";
+ }
+
+ @Override
+ public void close() throws IOException {
+ closeCurrentReader();
+ nextReady = null;
+ pendingRequests.clear();
+ pendingSearchIndex = Long.MIN_VALUE;
+ pendingLocalSeq = Long.MIN_VALUE;
+ lastError = null;
+ incompleteScan = false;
+ incompleteScanDetail = null;
+ resetCurrentFileTracking();
+ }
+
+ private IndexedConsensusRequest advance() throws IOException {
+ while (true) {
+ if (currentReader != null && currentReader.hasNext()) {
+ try {
+ final ByteBuffer buffer = currentReader.next();
+ consumedEntryCountInCurrentFile = currentReader.getCurrentEntryIndex() + 1;
+ final WALEntryType type = WALEntryType.valueOf(buffer.get());
+ buffer.clear();
+ if (!type.needSearch()) {
+ continue;
+ }
+
+ final long localSeq = currentReader.getCurrentEntryLocalSeq();
+ final long physicalTime = currentReader.getCurrentEntryPhysicalTime();
+ final int nodeId = currentReader.getCurrentEntryNodeId();
+
+ buffer.position(SEARCH_INDEX_OFFSET);
+ final long bodySearchIndex = buffer.getLong();
+ buffer.clear();
+
+ if (isSamePendingRequest(physicalTime, nodeId, localSeq)) {
+ if (pendingSearchIndex < 0 && bodySearchIndex >= 0) {
+ pendingSearchIndex = bodySearchIndex;
+ }
+ pendingRequests.add(new IoTConsensusRequest(buffer));
+ continue;
+ }
+
+ final IndexedConsensusRequest flushed = flushPending();
+ startPending(bodySearchIndex, physicalTime, nodeId, localSeq, buffer);
+ if (flushed != null && !shouldSkip(flushed)) {
+ return flushed;
+ }
+ continue;
+ } catch (final EOFException eofException) {
+ if (!currentReaderUsesLiveSnapshot) {
+ throw eofException;
+ }
+ // Live snapshot metadata may get ahead of the bytes currently visible in the file. Treat
+ // EOF as "this snapshot is exhausted for now" instead of terminating the iterator.
+ final IndexedConsensusRequest flushed = flushPending();
+ if (flushed != null && !shouldSkip(flushed)) {
+ closeCurrentReader();
+ return flushed;
+ }
+ if (reopenLiveSnapshotReader()) {
+ continue;
+ }
+ return null;
+ }
+ }
+
+ if (currentReaderUsesLiveSnapshot) {
+ final IndexedConsensusRequest flushed = flushPending();
+ if (flushed != null && !shouldSkip(flushed)) {
+ return flushed;
+ }
+ if (reopenLiveSnapshotReader()) {
+ continue;
+ }
+ return null;
+ }
+
+ if (currentReader != null) {
+ closeCurrentReader();
+ final IndexedConsensusRequest flushed = flushPending();
+ resetCurrentFileTracking();
+ if (flushed != null && !shouldSkip(flushed)) {
+ return flushed;
+ }
+ continue;
+ }
+
+ if (!openNextReader()) {
+ final IndexedConsensusRequest flushed = flushPending();
+ if (flushed != null && !shouldSkip(flushed)) {
+ return flushed;
+ }
+ return null;
+ }
+ }
+ }
+
+ private boolean openNextReader() throws IOException {
+ while (++currentFileIndex < walFiles.length) {
+ if (openReaderAtIndex(currentFileIndex, 0)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private boolean reopenLiveSnapshotReader() throws IOException {
+ if (liveWalNode == null || currentReaderVersionId < 0) {
+ return false;
+ }
+
+ closeCurrentReader();
+ refresh();
+
+ final long currentLiveVersionId = liveWalNode.getCurrentWALFileVersion();
+ if (currentLiveVersionId == currentReaderVersionId) {
+ final WALMetaData snapshot = liveWalNode.getCurrentWALMetaDataSnapshot();
+ if (snapshot.getBuffersSize().size() <= consumedEntryCountInCurrentFile) {
+ return false;
+ }
+ final int fileIndex = findFileIndexByVersion(currentReaderVersionId);
+ if (fileIndex < 0) {
+ return false;
+ }
+ return openReaderAtIndex(fileIndex, consumedEntryCountInCurrentFile);
+ }
+
+ final int previousFileIndex = findFileIndexByVersion(currentReaderVersionId);
+ if (previousFileIndex < 0) {
+ return openFirstReaderAfterVersion(currentReaderVersionId);
+ }
+ if (openReaderAtIndex(previousFileIndex, consumedEntryCountInCurrentFile)) {
+ return true;
+ }
+ return openFirstReaderAfterVersion(currentReaderVersionId);
+ }
+
+ private boolean openReaderAtIndex(final int fileIndex, final int skipEntries) throws IOException {
+ return openReaderAtIndex(fileIndex, skipEntries, true);
+ }
+
+ private boolean openReaderAtIndex(
+ final int fileIndex, final int skipEntries, final boolean allowNearLiveRetry)
+ throws IOException {
+ final File walFile = walFiles[fileIndex];
+ final long versionId = WALFileUtils.parseVersionId(walFile.getName());
+ final boolean useLiveSnapshot =
+ liveWalNode != null && versionId == liveWalNode.getCurrentWALFileVersion();
+
+ try {
+ final ProgressWALReader reader =
+ useLiveSnapshot
+ ? new ProgressWALReader(walFile, liveWalNode.getCurrentWALMetaDataSnapshot())
+ : new ProgressWALReader(walFile);
+ if (!skipEntries(reader, skipEntries)) {
+ reader.close();
+ markIncompleteScan(
+ String.format(
+ "failed to reopen WAL file %s at entry offset %s: iterator could not skip to the requested position",
+ walFile.getName(), skipEntries),
+ null);
+ resetCurrentFileTracking();
+ return false;
+ }
+ currentReader = reader;
+ currentFileIndex = fileIndex;
+ currentReaderVersionId = versionId;
+ currentReaderUsesLiveSnapshot = useLiveSnapshot;
+ consumedEntryCountInCurrentFile = skipEntries;
+ return true;
+ } catch (final IOException e) {
+ if (isNearLiveWalVersion(versionId)) {
+ LOGGER.debug(
+ "ProgressWALIterator: failed to open near-live WAL file {}, retrying without blacklisting",
+ walFile.getName(),
+ e);
+ if (allowNearLiveRetry) {
+ refresh();
+ final int refreshedIndex = findFileIndexByVersion(versionId);
+ if (refreshedIndex >= 0) {
+ if (openReaderAtIndex(refreshedIndex, skipEntries, false)) {
+ return true;
+ }
+ }
+ }
+ markIncompleteScan(
+ String.format(
+ "failed to open near-live WAL file %s while replay scan was still in progress",
+ walFile.getName()),
+ e);
+ return false;
+ }
+ skippedBrokenWalVersionIds.add(versionId);
+ LOGGER.warn(
+ "ProgressWALIterator: failed to open WAL file {}, skipping", walFile.getName(), e);
+ return false;
+ }
+ }
+
+ private boolean skipEntries(final ProgressWALReader reader, final int skipEntries)
+ throws IOException {
+ int skipped = 0;
+ while (skipped < skipEntries) {
+ if (!reader.hasNext()) {
+ return false;
+ }
+ reader.next();
+ skipped++;
+ }
+ return true;
+ }
Review Comment:
Pushed entry skipping down to the WAL reader by seeking to the metadata-derived logical offset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3345501697
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionPullConsumer.java:
##########
@@ -279,6 +303,139 @@ protected AbstractSubscriptionPullConsumer addProcessor(
return this;
}
+ @Override
+ List<SubscriptionCommitContext> getProcessorBufferedCommitContexts(final int dataNodeId) {
+ if (processorBufferedCommitContexts.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final List<SubscriptionCommitContext> result = new ArrayList<>();
+ for (final SubscriptionCommitContext commitContext : processorBufferedCommitContexts) {
+ if (Objects.nonNull(commitContext) && commitContext.getDataNodeId() == dataNodeId) {
+ result.add(commitContext);
+ }
+ }
+ return result;
+ }
+
+ private void refreshProcessorBufferedCommitContexts() {
+ processorBufferedCommitContexts.clear();
+ for (final SubscriptionMessageProcessor processor : processors) {
+ final List<SubscriptionCommitContext> bufferedCommitContexts =
+ processor.getBufferedCommitContexts();
+ if (Objects.isNull(bufferedCommitContexts)) {
+ continue;
+ }
+ for (final SubscriptionCommitContext commitContext : bufferedCommitContexts) {
+ if (Objects.nonNull(commitContext) && commitContext.isCommittable()) {
+ processorBufferedCommitContexts.add(commitContext);
+ }
+ }
+ }
+ }
+
+ private List<SubscriptionMessage> drainProcessorPipeline() {
+ List<SubscriptionMessage> drainedMessages = Collections.emptyList();
+ for (final SubscriptionMessageProcessor processor : processors) {
+ if (!drainedMessages.isEmpty()) {
+ drainedMessages = processor.process(drainedMessages);
+ if (Objects.isNull(drainedMessages)) {
+ drainedMessages = Collections.emptyList();
+ }
+ }
+
+ final List<SubscriptionMessage> flushedMessages = processor.flush();
+ if (Objects.nonNull(flushedMessages) && !flushedMessages.isEmpty()) {
+ if (drainedMessages.isEmpty()) {
+ drainedMessages = new ArrayList<>(flushedMessages);
+ } else {
+ drainedMessages = new ArrayList<>(drainedMessages);
+ drainedMessages.addAll(flushedMessages);
+ }
+ }
+ }
Review Comment:
Why is `drainedMessages` newed every time?
Will the processor modify the list after `flush()` is called?
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionProvider.java:
##########
@@ -526,6 +546,37 @@ void commit(final List<SubscriptionCommitContext> subscriptionCommitContexts, fi
throw new SubscriptionConnectionException(e.getMessage(), e);
}
verifyPipeSubscribeSuccess(resp.status);
+ final PipeSubscribeCommitResp commitResp = PipeSubscribeCommitResp.fromTPipeSubscribeResp(resp);
+ return new CommitResult(
+ commitResp.getAcceptedCommitContexts(), commitResp.getCommittedProgressByTopic());
+ }
+
+ static final class CommitResult {
+
+ private final List<SubscriptionCommitContext> acceptedCommitContexts;
+
+ private final Map<String, TopicProgress> committedProgressByTopic;
+
+ private CommitResult(
+ final List<SubscriptionCommitContext> acceptedCommitContexts,
+ final Map<String, TopicProgress> committedProgressByTopic) {
+ this.acceptedCommitContexts =
+ acceptedCommitContexts == null ? Collections.emptyList() : acceptedCommitContexts;
+ this.committedProgressByTopic =
+ committedProgressByTopic == null ? Collections.emptyMap() : committedProgressByTopic;
+ }
+
+ static CommitResult empty() {
+ return new CommitResult(Collections.emptyList(), Collections.emptyMap());
+ }
Review Comment:
May add a static member and reuse it, if this method is called often.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -397,22 +473,32 @@ private String buildUnsupportedConsensusRuntimeMessage(
}
public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
- final String consumerGroupId = commitContext.getConsumerGroupId();
- final String topicName = commitContext.getTopicName();
+ final ISubscriptionBroker broker =
+ getBrokerForTopic(commitContext.getConsumerGroupId(), commitContext.getTopicName());
+ return Objects.isNull(broker) || broker.isCommitContextOutdated(commitContext);
+ }
- // Try consensus broker first
+ private ISubscriptionBroker getBrokerForTopic(
+ final String consumerGroupId, final String topicName) {
final ConsensusSubscriptionBroker consensusBroker =
consumerGroupIdToConsensusBroker.get(consumerGroupId);
if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
- return consensusBroker.isCommitContextOutdated(commitContext);
+ return consensusBroker;
}
+ return consumerGroupIdToPipeBroker.get(consumerGroupId);
+ }
- // Fall back to pipe broker
- final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
- if (Objects.isNull(pipeBroker)) {
- return true;
+ private ISubscriptionBroker getBrokerForTopicOrThrow(
+ final String consumerGroupId, final String topicName) {
+ final ISubscriptionBroker broker = getBrokerForTopic(consumerGroupId, topicName);
+ if (Objects.nonNull(broker)) {
+ return broker;
}
- return pipeBroker.isCommitContextOutdated(commitContext);
+ final String errorMessage =
+ String.format(
+ "Subscription: broker bound to consumer group [%s] does not exist", consumerGroupId);
Review Comment:
i18n
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -507,16 +592,13 @@ public boolean dropBroker(final String consumerGroupId) {
public void bindPrefetchingQueue(final SubscriptionSinkSubtask subtask) {
final String consumerGroupId = subtask.getConsumerGroupId();
consumerGroupIdToPipeBroker
- .compute(
+ .computeIfAbsent(
consumerGroupId,
- (id, broker) -> {
- if (Objects.isNull(broker)) {
- LOGGER.info(
- "Subscription: pipe broker bound to consumer group [{}] does not exist, create new for binding prefetching queue",
- consumerGroupId);
- return new SubscriptionBroker(consumerGroupId);
- }
- return broker;
+ id -> {
+ LOGGER.info(
+ "Subscription: pipe broker bound to consumer group [{}] does not exist, create new for binding prefetching queue",
+ consumerGroupId);
Review Comment:
i18n
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessor.java:
##########
@@ -63,39 +70,72 @@ public List<SubscriptionMessage> flush() {
return Collections.emptyList();
}
- private void fillTablet(final Tablet tablet) {
- final String deviceKey = getDeviceKey(tablet);
- final Map<String, Object> cache = lastValues.computeIfAbsent(deviceKey, k -> new HashMap<>());
+ @Override
+ public void reset() {
+ lastValues.clear();
+ }
+
+ @Override
+ public boolean supportsTopicScopedReset() {
+ return true;
+ }
+
+ @Override
+ public void reset(final String topicName) {
+ lastValues.remove(topicName);
+ }
+
+ private void fillTablet(final String topicName, final Tablet tablet) {
+ final Map<String, Map<String, Pair<Long, Object>>> topicCache =
+ lastValues.computeIfAbsent(topicName, ignored -> new HashMap<>());
final Object[] values = tablet.getValues();
final BitMap[] bitMaps = tablet.getBitMaps();
final int rowSize = tablet.getRowSize();
final int columnCount = values.length;
for (int row = 0; row < rowSize; row++) {
+ final long timestamp = tablet.getTimestamp(row);
+ final String deviceKey = getDeviceKey(tablet, row);
+ if (deviceKey == null) {
+ continue;
+ }
+ final Map<String, Pair<Long, Object>> cache =
+ topicCache.computeIfAbsent(deviceKey, ignored -> new HashMap<>());
for (int col = 0; col < columnCount; col++) {
final String columnKey = getColumnKey(tablet, col);
final boolean isNull =
bitMaps != null && bitMaps[col] != null && bitMaps[col].isMarked(row);
if (isNull) {
- // try forward-fill from cache
- final Object cached = cache.get(columnKey);
- if (cached != null) {
- setValueAt(values[col], row, cached);
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached != null && cached.left < timestamp) {
+ setValueAt(values[col], row, cached.right);
bitMaps[col].unmark(row);
}
} else {
- // update cache with this non-null value
- cache.put(columnKey, getValueAt(values[col], row));
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached == null || timestamp >= cached.left) {
+ cache.put(columnKey, new Pair<>(timestamp, getValueAt(values[col], row)));
+ }
Review Comment:
If cached != null, may directly update the cache?
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) throws IOException {
+ final PipeSubscribeHeartbeatReq req = toTPipeSubscribeReq();
+ req.processorBufferedCommitContexts =
+ Objects.nonNull(processorBufferedCommitContexts)
+ ? processorBufferedCommitContexts
+ : new ArrayList<>();
Review Comment:
Collections.emptyList()?
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeHeartbeatReq.java:
##########
@@ -40,11 +56,43 @@ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq() {
return req;
}
+ /**
+ * Serialize the incoming parameters into `PipeSubscribeHeartbeatReq`, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeHeartbeatReq toTPipeSubscribeReq(
Review Comment:
Subscription is not restricted to Pipe now; may remove `Pipe` from the associated names.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -238,42 +225,136 @@ public List<SubscriptionCommitContext> commit(
return allSuccessful;
}
- public void seek(
- final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
- final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ public int refreshInFlightEventLeases(
+ final ConsumerConfig consumerConfig,
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) {
+ if (Objects.isNull(processorBufferedCommitContexts)
+ || processorBufferedCommitContexts.isEmpty()) {
+ return 0;
+ }
- final ConsensusSubscriptionBroker consensusBroker =
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ final String consumerId = consumerConfig.getConsumerId();
+ final ISubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ISubscriptionBroker consensusBroker =
consumerGroupIdToConsensusBroker.get(consumerGroupId);
Review Comment:
Is it possible to store both in the same map?
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessor.java:
##########
@@ -63,39 +70,72 @@ public List<SubscriptionMessage> flush() {
return Collections.emptyList();
}
- private void fillTablet(final Tablet tablet) {
- final String deviceKey = getDeviceKey(tablet);
- final Map<String, Object> cache = lastValues.computeIfAbsent(deviceKey, k -> new HashMap<>());
+ @Override
+ public void reset() {
+ lastValues.clear();
+ }
+
+ @Override
+ public boolean supportsTopicScopedReset() {
+ return true;
+ }
+
+ @Override
+ public void reset(final String topicName) {
+ lastValues.remove(topicName);
+ }
+
+ private void fillTablet(final String topicName, final Tablet tablet) {
+ final Map<String, Map<String, Pair<Long, Object>>> topicCache =
+ lastValues.computeIfAbsent(topicName, ignored -> new HashMap<>());
final Object[] values = tablet.getValues();
final BitMap[] bitMaps = tablet.getBitMaps();
final int rowSize = tablet.getRowSize();
final int columnCount = values.length;
for (int row = 0; row < rowSize; row++) {
+ final long timestamp = tablet.getTimestamp(row);
+ final String deviceKey = getDeviceKey(tablet, row);
+ if (deviceKey == null) {
+ continue;
+ }
+ final Map<String, Pair<Long, Object>> cache =
+ topicCache.computeIfAbsent(deviceKey, ignored -> new HashMap<>());
for (int col = 0; col < columnCount; col++) {
final String columnKey = getColumnKey(tablet, col);
final boolean isNull =
bitMaps != null && bitMaps[col] != null && bitMaps[col].isMarked(row);
if (isNull) {
- // try forward-fill from cache
- final Object cached = cache.get(columnKey);
- if (cached != null) {
- setValueAt(values[col], row, cached);
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached != null && cached.left < timestamp) {
+ setValueAt(values[col], row, cached.right);
bitMaps[col].unmark(row);
}
} else {
- // update cache with this non-null value
- cache.put(columnKey, getValueAt(values[col], row));
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached == null || timestamp >= cached.left) {
+ cache.put(columnKey, new Pair<>(timestamp, getValueAt(values[col], row)));
+ }
}
}
}
}
- private static String getDeviceKey(final Tablet tablet) {
- // tree model uses deviceId; table model uses tableName
+ private static String getDeviceKey(final Tablet tablet, final int row) {
+ if (hasTableDeviceIdentity(tablet)) {
+ return "table:" + tablet.getDeviceID(row);
+ }
+
final String deviceId = tablet.getDeviceId();
- return deviceId != null ? deviceId : tablet.getTableName();
+ return deviceId != null && deviceId.startsWith(TREE_MODEL_DEVICE_PREFIX)
+ ? "tree:" + deviceId
+ : null;
+ }
+
+ private static boolean hasTableDeviceIdentity(final Tablet tablet) {
+ final List<ColumnCategory> columnTypes = tablet.getColumnTypes();
+ return columnTypes != null && columnTypes.stream().anyMatch(type -> type == ColumnCategory.TAG);
Review Comment:
Is it possible to pass the dialect to the processor during construction?
It is invalid to distinguish a table or a tree like this, since a table may not have a tag column.
May directly use the IDeviceId as the map key, because two devices may be represented by the same string.
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/service/IoTConsensusRPCServiceProcessor.java:
##########
Review Comment:
Use i18n
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -529,33 +570,210 @@ public void receiveProgressBroadcast(
// ======================== Helper Methods ========================
- // Use a separator that cannot appear in consumerGroupId, topicName, or regionId
- // to prevent key collisions (e.g., "a_b" + "c" vs "a" + "b_c").
+ // Kept as the in-memory and ConfigNode sync key separator for the existing progress protocol.
private static final String KEY_SEPARATOR = "##";
private String generateKey(
final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
- return consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionId.toString();
+ return generateKey(consumerGroupId, topicName, regionId.toString());
}
- private File getProgressFile(final String key) {
- return new File(persistDir, PROGRESS_FILE_PREFIX + key + PROGRESS_FILE_SUFFIX);
+ private String generateKey(
+ final String consumerGroupId, final String topicName, final String regionIdStr) {
+ return consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionIdStr;
}
- private ConsensusSubscriptionCommitState tryRecover(final String key, final String regionIdStr) {
- final File file = getProgressFile(key);
- if (!file.exists()) {
+ private CommitStateKey getCommitStateKey(
+ final String consumerGroupId, final String topicName, final String regionIdStr) {
+ final String stateKey = generateKey(consumerGroupId, topicName, regionIdStr);
+ return commitStateKeys.computeIfAbsent(
+ stateKey,
+ ignored ->
+ new CommitStateKey(
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ stateKey,
+ generateTopicFileKey(consumerGroupId, topicName)));
+ }
+
+ private String generateTopicFileKey(final String consumerGroupId, final String topicName) {
+ return encodeFileNameComponent(consumerGroupId)
+ + KEY_SEPARATOR
+ + encodeFileNameComponent(topicName);
+ }
+
+ private static String encodeFileNameComponent(final String value) {
+ return Base64.getUrlEncoder()
+ .withoutPadding()
+ .encodeToString(String.valueOf(value).getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static String decodeFileNameComponent(final String value) {
+ switch (value.length() & 3) {
+ case 0:
+ return new String(Base64.getUrlDecoder().decode(value), StandardCharsets.UTF_8);
+ case 2:
+ return new String(Base64.getUrlDecoder().decode(value + "=="), StandardCharsets.UTF_8);
+ case 3:
+ return new String(Base64.getUrlDecoder().decode(value + "="), StandardCharsets.UTF_8);
+ default:
+ throw new IllegalArgumentException("Invalid base64 url component length");
+ }
+ }
+
+ private static String[] decodeTopicFileKey(final String topicFileKey) {
+ final int separatorIndex = topicFileKey.indexOf(KEY_SEPARATOR);
+ if (separatorIndex < 0) {
return null;
}
- try (final FileInputStream fis = new FileInputStream(file)) {
- final byte[] bytes = new byte[(int) file.length()];
- fis.read(bytes);
- final ByteBuffer buffer = ByteBuffer.wrap(bytes);
- return ConsensusSubscriptionCommitState.deserialize(regionIdStr, buffer);
- } catch (final IOException e) {
- LOGGER.warn("Failed to recover consensus subscription progress from {}", file, e);
+ try {
+ return new String[] {
+ decodeFileNameComponent(topicFileKey.substring(0, separatorIndex)),
+ decodeFileNameComponent(topicFileKey.substring(separatorIndex + KEY_SEPARATOR.length()))
+ };
+ } catch (final IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ private Object getTopicPersistLock(final String topicFileKey) {
+ return topicPersistLocks.computeIfAbsent(topicFileKey, ignored -> new Object());
+ }
+
+ private File getMetaFile(final String topicFileKey) {
+ return new File(persistDir, PROGRESS_FILE_PREFIX + topicFileKey + PROGRESS_META_FILE_SUFFIX);
+ }
+
+ private File getLegacyIndexFile(final String topicFileKey) {
+ return new File(
+ persistDir, PROGRESS_FILE_PREFIX + topicFileKey + LEGACY_PROGRESS_INDEX_FILE_SUFFIX);
+ }
+
+ private void recoverAllTopicStatesIfNeeded() {
+ final File dir = new File(persistDir);
+ final File[] metaFiles =
+ dir.listFiles(
+ (ignored, name) ->
+ name.startsWith(PROGRESS_FILE_PREFIX) && name.endsWith(PROGRESS_META_FILE_SUFFIX));
+ if (Objects.isNull(metaFiles)) {
+ return;
+ }
+ for (final File metaFile : metaFiles) {
+ final String topicFileKey = extractTopicFileKey(metaFile.getName());
+ if (Objects.isNull(topicFileKey) || recoveredTopicKeys.contains(topicFileKey)) {
+ continue;
+ }
+ final String[] decodedTopicKey = decodeTopicFileKey(topicFileKey);
+ if (Objects.isNull(decodedTopicKey)) {
+ LOGGER.warn(
+ "Skip malformed consensus subscription progress file name {}", metaFile.getName());
+ continue;
+ }
+ recoverTopicStatesIfNeeded(topicFileKey, decodedTopicKey[0], decodedTopicKey[1]);
+ }
+ }
+
+ private String extractTopicFileKey(final String metaFileName) {
+ if (!metaFileName.startsWith(PROGRESS_FILE_PREFIX)
+ || !metaFileName.endsWith(PROGRESS_META_FILE_SUFFIX)) {
return null;
}
+ return metaFileName.substring(
+ PROGRESS_FILE_PREFIX.length(), metaFileName.length() - PROGRESS_META_FILE_SUFFIX.length());
+ }
+
+ private void recoverTopicStatesIfNeeded(final CommitStateKey stateKey) {
+ recoverTopicStatesIfNeeded(stateKey.topicFileKey, stateKey.consumerGroupId, stateKey.topicName);
+ }
+
+ private void recoverTopicStatesIfNeeded(
+ final String topicFileKey, final String consumerGroupId, final String topicName) {
+ if (recoveredTopicKeys.contains(topicFileKey)) {
+ return;
+ }
+ synchronized (getTopicPersistLock(topicFileKey)) {
+ if (recoveredTopicKeys.contains(topicFileKey)) {
+ return;
+ }
+ try {
+ final TopicProgressSnapshot snapshot = readTopicProgressSnapshot(topicFileKey);
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ snapshot.states.entrySet()) {
+ final CommitStateKey recoveredStateKey =
+ getCommitStateKey(consumerGroupId, topicName, entry.getKey());
+ commitStates.putIfAbsent(recoveredStateKey.stateKey, entry.getValue());
+ }
+ topicProgressIndexes.put(topicFileKey, snapshot.indexEntries);
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "Failed to recover consensus subscription progress for consumerGroupId={}, "
+ + "topicName={}",
+ consumerGroupId,
+ topicName,
+ e);
+ topicProgressIndexes.put(topicFileKey, Collections.emptyMap());
+ } finally {
+ recoveredTopicKeys.add(topicFileKey);
+ }
+ }
+ }
+
+ // Recovery reads .meta sequentially and rebuilds the runtime offset index.
+ private TopicProgressSnapshot readTopicProgressSnapshot(final String topicFileKey)
+ throws IOException {
+ final File metaFile = getMetaFile(topicFileKey);
+ if (!metaFile.exists()) {
+ return TopicProgressSnapshot.empty();
+ }
+ try {
+ final ByteBuffer buffer = ByteBuffer.wrap(Files.readAllBytes(metaFile.toPath()));
+ final int version = ReadWriteIOUtils.readInt(buffer);
+ if (version != TOPIC_PROGRESS_FILE_VERSION) {
+ throw new IOException(
+ "Unsupported consensus subscription progress file version " + version);
Review Comment:
i18n
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -238,42 +225,136 @@ public List<SubscriptionCommitContext> commit(
return allSuccessful;
}
- public void seek(
- final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
- final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ public int refreshInFlightEventLeases(
+ final ConsumerConfig consumerConfig,
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) {
+ if (Objects.isNull(processorBufferedCommitContexts)
+ || processorBufferedCommitContexts.isEmpty()) {
+ return 0;
+ }
- final ConsensusSubscriptionBroker consensusBroker =
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ final String consumerId = consumerConfig.getConsumerId();
+ final ISubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ISubscriptionBroker consensusBroker =
consumerGroupIdToConsensusBroker.get(consumerGroupId);
- if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
- ensureConsensusSeekRuntimeAvailable(consumerGroupId, topicName, "seek");
- if (seekType != PipeSubscribeSeekReq.SEEK_TO_BEGINNING
- && seekType != PipeSubscribeSeekReq.SEEK_TO_END) {
- final String errorMessage =
- String.format(
- "Subscription: consensus seek only supports beginning/end or topic progress, "
- + "consumerGroup=%s, topic=%s, seekType=%s",
- consumerGroupId, topicName, seekType);
- LOGGER.warn(errorMessage);
- throw new SubscriptionException(errorMessage);
+
+ final List<SubscriptionCommitContext> pipeContexts = new ArrayList<>();
+ final List<SubscriptionCommitContext> consensusContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext ctx : processorBufferedCommitContexts) {
+ if (Objects.isNull(ctx) || Objects.isNull(ctx.getTopicName())) {
+ continue;
}
- consensusBroker.seek(topicName, seekType);
+ if (Objects.nonNull(consensusBroker)
+ && ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(ctx.getTopicName())) {
+ consensusContexts.add(ctx);
+ } else {
+ pipeContexts.add(ctx);
+ }
+ }
+
+ int refreshedCount = 0;
+ if (Objects.nonNull(pipeBroker) && !pipeContexts.isEmpty()) {
+ refreshedCount += pipeBroker.refreshInFlightEventLeases(consumerId, pipeContexts);
+ }
+ if (Objects.nonNull(consensusBroker) && !consensusContexts.isEmpty()) {
+ refreshedCount += consensusBroker.refreshInFlightEventLeases(consumerId, consensusContexts);
+ }
+ return refreshedCount;
+ }
+
+ public Map<String, TopicProgress> getConsensusCommittedProgressByTopic(
+ final ConsumerConfig consumerConfig,
+ final List<SubscriptionCommitContext> acceptedCommitContexts,
+ final boolean nack) {
+ if (nack || acceptedCommitContexts.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ final Map<String, Set<String>> regionIdsByTopic = new LinkedHashMap<>();
+ for (final SubscriptionCommitContext commitContext : acceptedCommitContexts) {
+ if (Objects.isNull(commitContext) || Objects.isNull(commitContext.getTopicName())) {
+ continue;
+ }
+ final String topicName = commitContext.getTopicName();
+ if (!ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName)) {
+ continue;
+ }
+ final String regionId = commitContext.getRegionId();
+ if (Objects.isNull(regionId) || regionId.isEmpty()) {
+ continue;
+ }
+ regionIdsByTopic.computeIfAbsent(topicName, ignored -> new LinkedHashSet<>()).add(regionId);
+ }
+
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ final Map<String, TopicProgress> result = new LinkedHashMap<>();
+ for (final Map.Entry<String, Set<String>> entry : regionIdsByTopic.entrySet()) {
+ final String topicName = entry.getKey();
+ final Map<String, RegionProgress> regionProgressById = new LinkedHashMap<>();
+ for (final String regionId : entry.getValue()) {
+ putRegionProgress(
+ regionProgressById,
+ regionId,
+ getConsensusCommittedRegionProgress(consumerGroupId, topicName, regionId));
+ }
+ if (!regionProgressById.isEmpty()) {
+ result.put(topicName, new TopicProgress(regionProgressById));
+ }
+ }
+ return result;
+ }
+
+ private RegionProgress getConsensusCommittedRegionProgress(
+ final String consumerGroupId, final String topicName, final String regionId) {
+ try {
+ return ConsensusSubscriptionCommitManager.getInstance()
+ .getCommittedRegionProgress(
+ consumerGroupId, topicName, ConsensusGroupId.Factory.createFromString(regionId));
+ } catch (final IllegalArgumentException e) {
+ LOGGER.warn(
+ "Subscription: failed to parse consensus region id {} for committed progress, topic={}, consumerGroup={}",
+ regionId,
+ topicName,
+ consumerGroupId,
+ e);
+ return null;
+ }
+ }
+
+ private void putRegionProgress(
+ final Map<String, RegionProgress> regionProgressById,
+ final String regionId,
+ final RegionProgress regionProgress) {
+ if (Objects.isNull(regionId)
+ || regionId.isEmpty()
+ || Objects.isNull(regionProgress)
+ || regionProgress.getWriterPositions().isEmpty()) {
return;
}
+ regionProgressById.put(regionId, regionProgress);
+ }
- if (isConsensusRuntimeUnsupported(topicName)) {
+ public void seek(
+ final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
+ if (seekType != SubscriptionSeekReq.SEEK_TO_BEGINNING
+ && seekType != SubscriptionSeekReq.SEEK_TO_END) {
final String errorMessage =
- buildUnsupportedConsensusRuntimeMessage(consumerGroupId, topicName, "seek");
+ String.format(
+ "Subscription: consensus seek only supports beginning/end or topic progress, "
+ + "consumerGroup=%s, topic=%s, seekType=%s",
+ consumerGroupId, topicName, seekType);
LOGGER.warn(errorMessage);
throw new SubscriptionException(errorMessage);
}
- final String errorMessage =
- String.format(
- "Subscription: seek is only supported for consensus-based subscriptions, "
- + "consumerGroup=%s, topic=%s",
- consumerGroupId, topicName);
- LOGGER.warn(errorMessage);
- throw new SubscriptionException(errorMessage);
+ final ConsensusSubscriptionBroker consensusBroker =
+ getConsensusBrokerForSeekOrNoOp(consumerGroupId, topicName, "seek");
+ if (Objects.nonNull(consensusBroker)) {
+ consensusBroker.seek(topicName, seekType);
+ }
}
Review Comment:
Is it possible to implement the logic (accepting or rejecting) inside each broker.
Then the agent does not need to know which broker it is any more.
##########
iotdb-client/subscription/src/test/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessorTest.java:
##########
@@ -80,6 +82,73 @@ public void testNewColumnDoesNotReuseOldIndexCache() {
Assert.assertTrue(schemaChangedTablet.getBitMaps()[1].isMarked(0));
}
+ @Test
+ public void testTableModelForwardFillDoesNotCrossDevices() {
+ final ColumnAlignProcessor processor = new ColumnAlignProcessor();
+
+ processor.process(
+ Collections.singletonList(
+ messageForTablet(
+ tableModelTablet(new long[] {1L}, new String[] {"id1"}, new Long[] {11L}, false))));
+
+ final Tablet differentDeviceTablet =
+ tableModelTablet(new long[] {2L}, new String[] {"id2"}, new Long[] {0L}, true);
+
+ processor.process(Collections.singletonList(messageForTablet(differentDeviceTablet)));
+
+ Assert.assertTrue(differentDeviceTablet.getBitMaps()[1].isMarked(0));
+
+ final Tablet sameDeviceTablet =
+ tableModelTablet(new long[] {3L}, new String[] {"id1"}, new Long[] {0L}, true);
+
+ processor.process(Collections.singletonList(messageForTablet(sameDeviceTablet)));
+
+ Assert.assertEquals(11L, ((long[]) sameDeviceTablet.getValues()[1])[0]);
+ Assert.assertFalse(sameDeviceTablet.getBitMaps()[1].isMarked(0));
+ }
+
+ @Test
+ public void testForwardFillOnlyUsesEarlierTimestamp() {
+ final ColumnAlignProcessor processor = new ColumnAlignProcessor();
+
+ processor.process(
+ Collections.singletonList(
+ messageForTablet(tabletWithSingleRow(new String[] {"s1"}, new long[] {100L}, 10L))));
+
+ final Tablet olderNullTablet =
+ tabletWithSingleRow(new String[] {"s1"}, new long[] {0L}, 5L, true);
+
+ processor.process(Collections.singletonList(messageForTablet(olderNullTablet)));
+
+ Assert.assertTrue(olderNullTablet.getBitMaps()[0].isMarked(0));
+
+ processor.process(
+ Collections.singletonList(
+ messageForTablet(tabletWithSingleRow(new String[] {"s1"}, new long[] {50L}, 6L))));
+
+ final Tablet newerNullTablet =
+ tabletWithSingleRow(new String[] {"s1"}, new long[] {0L}, 11L, true);
+
+ processor.process(Collections.singletonList(messageForTablet(newerNullTablet)));
+
+ Assert.assertEquals(100L, ((long[]) newerNullTablet.getValues()[0])[0]);
+ Assert.assertFalse(newerNullTablet.getBitMaps()[0].isMarked(0));
+ }
+
+ @Test
+ public void testTableModelWithoutDeviceIdentityDoesNotForwardFill() {
+ final ColumnAlignProcessor processor = new ColumnAlignProcessor();
+
+ processor.process(
+ Collections.singletonList(messageForTablet(fieldOnlyTableModelTablet(1L, 11L, false))));
+
+ final Tablet ambiguousTablet = fieldOnlyTableModelTablet(2L, 0L, true);
+
+ processor.process(Collections.singletonList(messageForTablet(ambiguousTablet)));
+
+ Assert.assertTrue(ambiguousTablet.getBitMaps()[0].isMarked(0));
+ }
Review Comment:
The no-tag device (or all-tag-null device) is also a specific device that should be filled.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -146,16 +165,20 @@ private ConsensusSubscriptionCommitManager() {
*/
public ConsensusSubscriptionCommitState getOrCreateState(
final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
- final String key = generateKey(consumerGroupId, topicName, regionId);
final String regionIdString = regionId.toString();
+ final CommitStateKey stateKey = getCommitStateKey(consumerGroupId, topicName, regionIdString);
+ final ConsensusSubscriptionCommitState existing = commitStates.get(stateKey.stateKey);
Review Comment:
`stateKey.stateKey` is weird, rename one of them.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -110,17 +122,24 @@ public class ConsensusSubscriptionCommitManager {
/** Single-threaded executor for fire-and-forget broadcasts. */
private final ExecutorService broadcastExecutor =
- Executors.newSingleThreadExecutor(
- r -> {
- final Thread t = new Thread(r, "SubscriptionProgressBroadcast");
- t.setDaemon(true);
- return t;
- });
+ IoTDBThreadPoolFactory.newSingleThreadExecutor(
+ ThreadName.SUBSCRIPTION_CONSENSUS_PROGRESS_BROADCASTER.getName());
/** Key: "consumerGroupId##topicName##regionId" -> progress tracking state */
private final Map<String, ConsensusSubscriptionCommitState> commitStates =
new ConcurrentHashMap<>();
+ private final Map<String, CommitStateKey> commitStateKeys = new ConcurrentHashMap<>();
+
+ private final Set<String> recoveredTopicKeys = ConcurrentHashMap.newKeySet();
+
+ private final Map<String, Object> topicPersistLocks = new ConcurrentHashMap<>();
+
+ // Runtime random-write index. The on-disk .meta file is the only source of truth; this map is
+ // rebuilt from .meta during recovery and is never persisted separately.
+ private final Map<String, Map<String, ProgressIndexEntry>> topicProgressIndexes =
+ new ConcurrentHashMap<>();
+
Review Comment:
Exaplain the key of `commitStateKeys` and the second key of `topicProgressIndexes`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -1129,7 +1125,40 @@ public SubscriptionEvent pollTablets(
"ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
this, consumerId, commitContext));
}
- return event;
+ if (Objects.isNull(event.getCurrentResponse())
+ || event.getCurrentResponse().getResponseType()
+ != SubscriptionPollResponseType.TABLETS.getType()
+ || !(event.getCurrentResponse().getPayload() instanceof TabletsPayload)) {
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: unexpected in-flight response for consumer %s, "
+ + "commit context %s, offset %s",
Review Comment:
i18n?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertTabletNode;
+
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/** Converts IoTConsensus WAL log entries (InsertNode) to Tablet format for subscription. */
+public class ConsensusLogToTabletConverter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusLogToTabletConverter.class);
+
+ private final TreePattern treePattern;
+ private final TablePattern tablePattern;
+ private final Pattern tableColumnPattern;
+
+ /**
+ * The actual database name of the DataRegion this converter processes (table-model format without
+ * "root." prefix). Null for tree-model topics.
+ */
+ private final String databaseName;
+
+ public ConsensusLogToTabletConverter(
+ final TreePattern treePattern,
+ final TablePattern tablePattern,
+ final Pattern tableColumnPattern,
+ final String databaseName) {
+ this.treePattern = treePattern;
+ this.tablePattern = tablePattern;
+ this.tableColumnPattern = tableColumnPattern;
+ this.databaseName = databaseName;
+ }
+
+ public String getDatabaseName() {
+ return databaseName;
+ }
+
+ static String safeDeviceIdForLog(final InsertNode node) {
+ try {
+ final Object deviceId = node.getDeviceID();
+ return deviceId != null ? deviceId.toString() : "null";
+ } catch (final Exception e) {
+ return "N/A(" + node.getType() + ")";
+ }
+ }
+
+ public List<Tablet> convert(final InsertNode insertNode) {
+ if (Objects.isNull(insertNode)) {
+ return Collections.emptyList();
+ }
+
+ final PlanNodeType nodeType = insertNode.getType();
+ if (nodeType == null) {
+ LOGGER.warn("InsertNode type is null, skipping conversion");
+ return Collections.emptyList();
+ }
+
+ LOGGER.debug(
+ "ConsensusLogToTabletConverter: converting InsertNode type={}, deviceId={}",
+ nodeType,
+ safeDeviceIdForLog(insertNode));
+
+ switch (nodeType) {
+ case INSERT_ROW:
+ return convertInsertRowNode((InsertRowNode) insertNode);
+ case INSERT_TABLET:
+ return convertInsertTabletNode((InsertTabletNode) insertNode);
+ case INSERT_ROWS:
+ return convertInsertRowsNode((InsertRowsNode) insertNode);
+ case INSERT_ROWS_OF_ONE_DEVICE:
+ return convertInsertRowsOfOneDeviceNode((InsertRowsOfOneDeviceNode) insertNode);
+ case INSERT_MULTI_TABLET:
+ return convertInsertMultiTabletsNode((InsertMultiTabletsNode) insertNode);
+ case RELATIONAL_INSERT_ROW:
+ return convertRelationalInsertRowNode((RelationalInsertRowNode) insertNode);
+ case RELATIONAL_INSERT_TABLET:
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) insertNode);
+ case RELATIONAL_INSERT_ROWS:
+ return convertRelationalInsertRowsNode((RelationalInsertRowsNode) insertNode);
+ default:
+ LOGGER.debug("Unsupported InsertNode type for subscription: {}", nodeType);
+ return Collections.emptyList();
+ }
+ }
+
+ // ======================== Tree Model Conversion ========================
+
+ private List<Tablet> convertInsertRowNode(final InsertRowNode node) {
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final long time = node.getTime();
+
+ // Determine which columns match the pattern
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final Object[] values = node.getValues();
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Build Tablet with matched columns
+ final int columnCount = matchedColumnIndices.size();
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ final Tablet tablet = new Tablet(deviceId.toString(), schemas, 1 /* maxRowNumber */);
+ tablet.addTimestamp(0, time);
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = matchedColumnIndices.get(i);
+ final Object value = values[originalColIdx];
+ if (value == null) {
+ if (tablet.getBitMaps() == null) {
+ tablet.initBitMaps();
+ }
+ tablet.getBitMaps()[i].mark(0);
+ } else {
+ addValueToTablet(tablet, 0, i, dataTypes[originalColIdx], value);
+ }
+ }
+ tablet.setRowSize(1);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
+ if (node instanceof RelationalInsertTabletNode) {
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) node);
+ }
+
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final long[] times = node.getTimes();
+ final Object[] columns = node.getColumns();
+ final BitMap[] bitMaps = node.getBitMaps();
+ final int rowCount = node.getRowCount();
+
+ // Column filtering
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final int columnCount = matchedColumnIndices.size();
+ final boolean allColumnsMatch = (columnCount == measurements.length);
+
+ // Build schemas (always needed)
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ // Build column arrays and bitmaps using bulk copy
+ final long[] newTimes = Arrays.copyOf(times, rowCount);
+ final Object[] newColumns = new Object[columnCount];
+ final BitMap[] newBitMaps = new BitMap[columnCount];
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = allColumnsMatch ? i : matchedColumnIndices.get(i);
+ newColumns[i] = copyColumnArray(dataTypes[originalColIdx], columns[originalColIdx], rowCount);
+ if (bitMaps != null && bitMaps[originalColIdx] != null) {
+ newBitMaps[i] = new BitMap(rowCount);
+ BitMap.copyOfRange(bitMaps[originalColIdx], 0, newBitMaps[i], 0, rowCount);
+ }
+ }
Review Comment:
This will consume considerable memory bandwidth when multiple subscriptions are present.
We would better figure out which paths may modify the content of a tablet and use a copy-on-write mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3346562437
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionProvider.java:
##########
@@ -526,6 +546,37 @@ void commit(final List<SubscriptionCommitContext> subscriptionCommitContexts, fi
throw new SubscriptionConnectionException(e.getMessage(), e);
}
verifyPipeSubscribeSuccess(resp.status);
+ final PipeSubscribeCommitResp commitResp = PipeSubscribeCommitResp.fromTPipeSubscribeResp(resp);
+ return new CommitResult(
+ commitResp.getAcceptedCommitContexts(), commitResp.getCommittedProgressByTopic());
+ }
+
+ static final class CommitResult {
+
+ private final List<SubscriptionCommitContext> acceptedCommitContexts;
+
+ private final Map<String, TopicProgress> committedProgressByTopic;
+
+ private CommitResult(
+ final List<SubscriptionCommitContext> acceptedCommitContexts,
+ final Map<String, TopicProgress> committedProgressByTopic) {
+ this.acceptedCommitContexts =
+ acceptedCommitContexts == null ? Collections.emptyList() : acceptedCommitContexts;
+ this.committedProgressByTopic =
+ committedProgressByTopic == null ? Collections.emptyMap() : committedProgressByTopic;
+ }
+
+ static CommitResult empty() {
+ return new CommitResult(Collections.emptyList(), Collections.emptyMap());
+ }
Review Comment:
Done in 5ea428025e. CommitResult.empty() now returns a static reusable EMPTY instance.
##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties:
##########
@@ -54,6 +54,12 @@ dn_data_region_consensus_port=10760
schema_replication_factor=1
data_replication_factor=1
+####################
+### Subscription Consensus Configuration
+####################
+
+# subscription_consensus_idle_safe_hlc_interval_ms=10000
+
Review Comment:
Done in 5ea428025e. This property is no longer added to iotdb-system.properties; it remains in the template instead.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -238,42 +225,136 @@ public List<SubscriptionCommitContext> commit(
return allSuccessful;
}
- public void seek(
- final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
- final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ public int refreshInFlightEventLeases(
+ final ConsumerConfig consumerConfig,
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) {
+ if (Objects.isNull(processorBufferedCommitContexts)
+ || processorBufferedCommitContexts.isEmpty()) {
+ return 0;
+ }
- final ConsensusSubscriptionBroker consensusBroker =
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ final String consumerId = consumerConfig.getConsumerId();
+ final ISubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ISubscriptionBroker consensusBroker =
consumerGroupIdToConsensusBroker.get(consumerGroupId);
- if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
- ensureConsensusSeekRuntimeAvailable(consumerGroupId, topicName, "seek");
- if (seekType != PipeSubscribeSeekReq.SEEK_TO_BEGINNING
- && seekType != PipeSubscribeSeekReq.SEEK_TO_END) {
- final String errorMessage =
- String.format(
- "Subscription: consensus seek only supports beginning/end or topic progress, "
- + "consumerGroup=%s, topic=%s, seekType=%s",
- consumerGroupId, topicName, seekType);
- LOGGER.warn(errorMessage);
- throw new SubscriptionException(errorMessage);
+
+ final List<SubscriptionCommitContext> pipeContexts = new ArrayList<>();
+ final List<SubscriptionCommitContext> consensusContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext ctx : processorBufferedCommitContexts) {
+ if (Objects.isNull(ctx) || Objects.isNull(ctx.getTopicName())) {
+ continue;
}
- consensusBroker.seek(topicName, seekType);
+ if (Objects.nonNull(consensusBroker)
+ && ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(ctx.getTopicName())) {
+ consensusContexts.add(ctx);
+ } else {
+ pipeContexts.add(ctx);
+ }
+ }
+
+ int refreshedCount = 0;
+ if (Objects.nonNull(pipeBroker) && !pipeContexts.isEmpty()) {
+ refreshedCount += pipeBroker.refreshInFlightEventLeases(consumerId, pipeContexts);
+ }
+ if (Objects.nonNull(consensusBroker) && !consensusContexts.isEmpty()) {
+ refreshedCount += consensusBroker.refreshInFlightEventLeases(consumerId, consensusContexts);
+ }
+ return refreshedCount;
+ }
+
+ public Map<String, TopicProgress> getConsensusCommittedProgressByTopic(
+ final ConsumerConfig consumerConfig,
+ final List<SubscriptionCommitContext> acceptedCommitContexts,
+ final boolean nack) {
+ if (nack || acceptedCommitContexts.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ final Map<String, Set<String>> regionIdsByTopic = new LinkedHashMap<>();
+ for (final SubscriptionCommitContext commitContext : acceptedCommitContexts) {
+ if (Objects.isNull(commitContext) || Objects.isNull(commitContext.getTopicName())) {
+ continue;
+ }
+ final String topicName = commitContext.getTopicName();
+ if (!ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName)) {
+ continue;
+ }
+ final String regionId = commitContext.getRegionId();
+ if (Objects.isNull(regionId) || regionId.isEmpty()) {
+ continue;
+ }
+ regionIdsByTopic.computeIfAbsent(topicName, ignored -> new LinkedHashSet<>()).add(regionId);
+ }
+
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ final Map<String, TopicProgress> result = new LinkedHashMap<>();
+ for (final Map.Entry<String, Set<String>> entry : regionIdsByTopic.entrySet()) {
+ final String topicName = entry.getKey();
+ final Map<String, RegionProgress> regionProgressById = new LinkedHashMap<>();
+ for (final String regionId : entry.getValue()) {
+ putRegionProgress(
+ regionProgressById,
+ regionId,
+ getConsensusCommittedRegionProgress(consumerGroupId, topicName, regionId));
+ }
+ if (!regionProgressById.isEmpty()) {
+ result.put(topicName, new TopicProgress(regionProgressById));
+ }
+ }
+ return result;
+ }
+
+ private RegionProgress getConsensusCommittedRegionProgress(
+ final String consumerGroupId, final String topicName, final String regionId) {
+ try {
+ return ConsensusSubscriptionCommitManager.getInstance()
+ .getCommittedRegionProgress(
+ consumerGroupId, topicName, ConsensusGroupId.Factory.createFromString(regionId));
+ } catch (final IllegalArgumentException e) {
+ LOGGER.warn(
+ "Subscription: failed to parse consensus region id {} for committed progress, topic={}, consumerGroup={}",
+ regionId,
+ topicName,
+ consumerGroupId,
+ e);
+ return null;
+ }
+ }
+
+ private void putRegionProgress(
+ final Map<String, RegionProgress> regionProgressById,
+ final String regionId,
+ final RegionProgress regionProgress) {
+ if (Objects.isNull(regionId)
+ || regionId.isEmpty()
+ || Objects.isNull(regionProgress)
+ || regionProgress.getWriterPositions().isEmpty()) {
return;
}
+ regionProgressById.put(regionId, regionProgress);
+ }
- if (isConsensusRuntimeUnsupported(topicName)) {
+ public void seek(
+ final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
+ if (seekType != SubscriptionSeekReq.SEEK_TO_BEGINNING
+ && seekType != SubscriptionSeekReq.SEEK_TO_END) {
final String errorMessage =
- buildUnsupportedConsensusRuntimeMessage(consumerGroupId, topicName, "seek");
+ String.format(
+ "Subscription: consensus seek only supports beginning/end or topic progress, "
+ + "consumerGroup=%s, topic=%s, seekType=%s",
+ consumerGroupId, topicName, seekType);
LOGGER.warn(errorMessage);
throw new SubscriptionException(errorMessage);
}
- final String errorMessage =
- String.format(
- "Subscription: seek is only supported for consensus-based subscriptions, "
- + "consumerGroup=%s, topic=%s",
- consumerGroupId, topicName);
- LOGGER.warn(errorMessage);
- throw new SubscriptionException(errorMessage);
+ final ConsensusSubscriptionBroker consensusBroker =
+ getConsensusBrokerForSeekOrNoOp(consumerGroupId, topicName, "seek");
+ if (Objects.nonNull(consensusBroker)) {
+ consensusBroker.seek(topicName, seekType);
+ }
}
Review Comment:
Done in aa6810e062. ISubscriptionBroker now exposes acceptsTopic()/selectAcceptedCommitContexts(), so the agent delegates accept/reject routing to each broker instead of hard-coding broker type decisions in commit/lease paths.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -507,16 +592,13 @@ public boolean dropBroker(final String consumerGroupId) {
public void bindPrefetchingQueue(final SubscriptionSinkSubtask subtask) {
final String consumerGroupId = subtask.getConsumerGroupId();
consumerGroupIdToPipeBroker
- .compute(
+ .computeIfAbsent(
consumerGroupId,
- (id, broker) -> {
- if (Objects.isNull(broker)) {
- LOGGER.info(
- "Subscription: pipe broker bound to consumer group [{}] does not exist, create new for binding prefetching queue",
- consumerGroupId);
- return new SubscriptionBroker(consumerGroupId);
- }
- return broker;
+ id -> {
+ LOGGER.info(
+ "Subscription: pipe broker bound to consumer group [{}] does not exist, create new for binding prefetching queue",
+ consumerGroupId);
Review Comment:
Done in 5ea428025e. The create/drop/non-empty broker log messages now use DataNodeMiscMessages en/zh constants.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessor.java:
##########
@@ -63,39 +70,72 @@ public List<SubscriptionMessage> flush() {
return Collections.emptyList();
}
- private void fillTablet(final Tablet tablet) {
- final String deviceKey = getDeviceKey(tablet);
- final Map<String, Object> cache = lastValues.computeIfAbsent(deviceKey, k -> new HashMap<>());
+ @Override
+ public void reset() {
+ lastValues.clear();
+ }
+
+ @Override
+ public boolean supportsTopicScopedReset() {
+ return true;
+ }
+
+ @Override
+ public void reset(final String topicName) {
+ lastValues.remove(topicName);
+ }
+
+ private void fillTablet(final String topicName, final Tablet tablet) {
+ final Map<String, Map<String, Pair<Long, Object>>> topicCache =
+ lastValues.computeIfAbsent(topicName, ignored -> new HashMap<>());
final Object[] values = tablet.getValues();
final BitMap[] bitMaps = tablet.getBitMaps();
final int rowSize = tablet.getRowSize();
final int columnCount = values.length;
for (int row = 0; row < rowSize; row++) {
+ final long timestamp = tablet.getTimestamp(row);
+ final String deviceKey = getDeviceKey(tablet, row);
+ if (deviceKey == null) {
+ continue;
+ }
+ final Map<String, Pair<Long, Object>> cache =
+ topicCache.computeIfAbsent(deviceKey, ignored -> new HashMap<>());
for (int col = 0; col < columnCount; col++) {
final String columnKey = getColumnKey(tablet, col);
final boolean isNull =
bitMaps != null && bitMaps[col] != null && bitMaps[col].isMarked(row);
if (isNull) {
- // try forward-fill from cache
- final Object cached = cache.get(columnKey);
- if (cached != null) {
- setValueAt(values[col], row, cached);
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached != null && cached.left < timestamp) {
+ setValueAt(values[col], row, cached.right);
bitMaps[col].unmark(row);
}
} else {
- // update cache with this non-null value
- cache.put(columnKey, getValueAt(values[col], row));
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached == null || timestamp >= cached.left) {
+ cache.put(columnKey, new Pair<>(timestamp, getValueAt(values[col], row)));
+ }
Review Comment:
Done in 5ea428025e. The cached Pair is now updated in place when an existing cache entry is reused.
##########
iotdb-client/subscription/src/test/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessorTest.java:
##########
@@ -80,6 +82,73 @@ public void testNewColumnDoesNotReuseOldIndexCache() {
Assert.assertTrue(schemaChangedTablet.getBitMaps()[1].isMarked(0));
}
+ @Test
+ public void testTableModelForwardFillDoesNotCrossDevices() {
+ final ColumnAlignProcessor processor = new ColumnAlignProcessor();
+
+ processor.process(
+ Collections.singletonList(
+ messageForTablet(
+ tableModelTablet(new long[] {1L}, new String[] {"id1"}, new Long[] {11L}, false))));
+
+ final Tablet differentDeviceTablet =
+ tableModelTablet(new long[] {2L}, new String[] {"id2"}, new Long[] {0L}, true);
+
+ processor.process(Collections.singletonList(messageForTablet(differentDeviceTablet)));
+
+ Assert.assertTrue(differentDeviceTablet.getBitMaps()[1].isMarked(0));
+
+ final Tablet sameDeviceTablet =
+ tableModelTablet(new long[] {3L}, new String[] {"id1"}, new Long[] {0L}, true);
+
+ processor.process(Collections.singletonList(messageForTablet(sameDeviceTablet)));
+
+ Assert.assertEquals(11L, ((long[]) sameDeviceTablet.getValues()[1])[0]);
+ Assert.assertFalse(sameDeviceTablet.getBitMaps()[1].isMarked(0));
+ }
+
+ @Test
+ public void testForwardFillOnlyUsesEarlierTimestamp() {
+ final ColumnAlignProcessor processor = new ColumnAlignProcessor();
+
+ processor.process(
+ Collections.singletonList(
+ messageForTablet(tabletWithSingleRow(new String[] {"s1"}, new long[] {100L}, 10L))));
+
+ final Tablet olderNullTablet =
+ tabletWithSingleRow(new String[] {"s1"}, new long[] {0L}, 5L, true);
+
+ processor.process(Collections.singletonList(messageForTablet(olderNullTablet)));
+
+ Assert.assertTrue(olderNullTablet.getBitMaps()[0].isMarked(0));
+
+ processor.process(
+ Collections.singletonList(
+ messageForTablet(tabletWithSingleRow(new String[] {"s1"}, new long[] {50L}, 6L))));
+
+ final Tablet newerNullTablet =
+ tabletWithSingleRow(new String[] {"s1"}, new long[] {0L}, 11L, true);
+
+ processor.process(Collections.singletonList(messageForTablet(newerNullTablet)));
+
+ Assert.assertEquals(100L, ((long[]) newerNullTablet.getValues()[0])[0]);
+ Assert.assertFalse(newerNullTablet.getBitMaps()[0].isMarked(0));
+ }
+
+ @Test
+ public void testTableModelWithoutDeviceIdentityDoesNotForwardFill() {
+ final ColumnAlignProcessor processor = new ColumnAlignProcessor();
+
+ processor.process(
+ Collections.singletonList(messageForTablet(fieldOnlyTableModelTablet(1L, 11L, false))));
+
+ final Tablet ambiguousTablet = fieldOnlyTableModelTablet(2L, 0L, true);
+
+ processor.process(Collections.singletonList(messageForTablet(ambiguousTablet)));
+
+ Assert.assertTrue(ambiguousTablet.getBitMaps()[0].isMarked(0));
+ }
Review Comment:
Done in 5ea428025e. Table-mode tests now cover both no-tag and all-tag-null devices, and those rows are forward-filled as the same specific device.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessor.java:
##########
@@ -63,39 +70,72 @@ public List<SubscriptionMessage> flush() {
return Collections.emptyList();
}
- private void fillTablet(final Tablet tablet) {
- final String deviceKey = getDeviceKey(tablet);
- final Map<String, Object> cache = lastValues.computeIfAbsent(deviceKey, k -> new HashMap<>());
+ @Override
+ public void reset() {
+ lastValues.clear();
+ }
+
+ @Override
+ public boolean supportsTopicScopedReset() {
+ return true;
+ }
+
+ @Override
+ public void reset(final String topicName) {
+ lastValues.remove(topicName);
+ }
+
+ private void fillTablet(final String topicName, final Tablet tablet) {
+ final Map<String, Map<String, Pair<Long, Object>>> topicCache =
+ lastValues.computeIfAbsent(topicName, ignored -> new HashMap<>());
final Object[] values = tablet.getValues();
final BitMap[] bitMaps = tablet.getBitMaps();
final int rowSize = tablet.getRowSize();
final int columnCount = values.length;
for (int row = 0; row < rowSize; row++) {
+ final long timestamp = tablet.getTimestamp(row);
+ final String deviceKey = getDeviceKey(tablet, row);
+ if (deviceKey == null) {
+ continue;
+ }
+ final Map<String, Pair<Long, Object>> cache =
+ topicCache.computeIfAbsent(deviceKey, ignored -> new HashMap<>());
for (int col = 0; col < columnCount; col++) {
final String columnKey = getColumnKey(tablet, col);
final boolean isNull =
bitMaps != null && bitMaps[col] != null && bitMaps[col].isMarked(row);
if (isNull) {
- // try forward-fill from cache
- final Object cached = cache.get(columnKey);
- if (cached != null) {
- setValueAt(values[col], row, cached);
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached != null && cached.left < timestamp) {
+ setValueAt(values[col], row, cached.right);
bitMaps[col].unmark(row);
}
} else {
- // update cache with this non-null value
- cache.put(columnKey, getValueAt(values[col], row));
+ final Pair<Long, Object> cached = cache.get(columnKey);
+ if (cached == null || timestamp >= cached.left) {
+ cache.put(columnKey, new Pair<>(timestamp, getValueAt(values[col], row)));
+ }
}
}
}
}
- private static String getDeviceKey(final Tablet tablet) {
- // tree model uses deviceId; table model uses tableName
+ private static String getDeviceKey(final Tablet tablet, final int row) {
+ if (hasTableDeviceIdentity(tablet)) {
+ return "table:" + tablet.getDeviceID(row);
+ }
+
final String deviceId = tablet.getDeviceId();
- return deviceId != null ? deviceId : tablet.getTableName();
+ return deviceId != null && deviceId.startsWith(TREE_MODEL_DEVICE_PREFIX)
+ ? "tree:" + deviceId
+ : null;
+ }
+
+ private static boolean hasTableDeviceIdentity(final Tablet tablet) {
+ final List<ColumnCategory> columnTypes = tablet.getColumnTypes();
+ return columnTypes != null && columnTypes.stream().anyMatch(type -> type == ColumnCategory.TAG);
Review Comment:
Done in 5ea428025e. ColumnAlignProcessor now takes the dialect in the constructor, removed tag-column auto detection, and table mode keys the cache by IDeviceID.
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionPullConsumer.java:
##########
@@ -279,6 +303,139 @@ protected AbstractSubscriptionPullConsumer addProcessor(
return this;
}
+ @Override
+ List<SubscriptionCommitContext> getProcessorBufferedCommitContexts(final int dataNodeId) {
+ if (processorBufferedCommitContexts.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final List<SubscriptionCommitContext> result = new ArrayList<>();
+ for (final SubscriptionCommitContext commitContext : processorBufferedCommitContexts) {
+ if (Objects.nonNull(commitContext) && commitContext.getDataNodeId() == dataNodeId) {
+ result.add(commitContext);
+ }
+ }
+ return result;
+ }
+
+ private void refreshProcessorBufferedCommitContexts() {
+ processorBufferedCommitContexts.clear();
+ for (final SubscriptionMessageProcessor processor : processors) {
+ final List<SubscriptionCommitContext> bufferedCommitContexts =
+ processor.getBufferedCommitContexts();
+ if (Objects.isNull(bufferedCommitContexts)) {
+ continue;
+ }
+ for (final SubscriptionCommitContext commitContext : bufferedCommitContexts) {
+ if (Objects.nonNull(commitContext) && commitContext.isCommittable()) {
+ processorBufferedCommitContexts.add(commitContext);
+ }
+ }
+ }
+ }
+
+ private List<SubscriptionMessage> drainProcessorPipeline() {
+ List<SubscriptionMessage> drainedMessages = Collections.emptyList();
+ for (final SubscriptionMessageProcessor processor : processors) {
+ if (!drainedMessages.isEmpty()) {
+ drainedMessages = processor.process(drainedMessages);
+ if (Objects.isNull(drainedMessages)) {
+ drainedMessages = Collections.emptyList();
+ }
+ }
+
+ final List<SubscriptionMessage> flushedMessages = processor.flush();
+ if (Objects.nonNull(flushedMessages) && !flushedMessages.isEmpty()) {
+ if (drainedMessages.isEmpty()) {
+ drainedMessages = new ArrayList<>(flushedMessages);
+ } else {
+ drainedMessages = new ArrayList<>(drainedMessages);
+ drainedMessages.addAll(flushedMessages);
+ }
+ }
+ }
Review Comment:
Done in 5ea428025e. The flush() contract now documents ownership transfer, and drainProcessorPipeline() avoids allocating a new base list unless it really needs to merge non-empty flush results.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -146,16 +165,20 @@ private ConsensusSubscriptionCommitManager() {
*/
public ConsensusSubscriptionCommitState getOrCreateState(
final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
- final String key = generateKey(consumerGroupId, topicName, regionId);
final String regionIdString = regionId.toString();
+ final CommitStateKey stateKey = getCommitStateKey(consumerGroupId, topicName, regionIdString);
+ final ConsensusSubscriptionCommitState existing = commitStates.get(stateKey.stateKey);
Review Comment:
Done in 5ea428025e. Renamed CommitStateKey.stateKey to encodedStateKey.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -110,17 +122,24 @@ public class ConsensusSubscriptionCommitManager {
/** Single-threaded executor for fire-and-forget broadcasts. */
private final ExecutorService broadcastExecutor =
- Executors.newSingleThreadExecutor(
- r -> {
- final Thread t = new Thread(r, "SubscriptionProgressBroadcast");
- t.setDaemon(true);
- return t;
- });
+ IoTDBThreadPoolFactory.newSingleThreadExecutor(
+ ThreadName.SUBSCRIPTION_CONSENSUS_PROGRESS_BROADCASTER.getName());
/** Key: "consumerGroupId##topicName##regionId" -> progress tracking state */
private final Map<String, ConsensusSubscriptionCommitState> commitStates =
new ConcurrentHashMap<>();
+ private final Map<String, CommitStateKey> commitStateKeys = new ConcurrentHashMap<>();
+
+ private final Set<String> recoveredTopicKeys = ConcurrentHashMap.newKeySet();
+
+ private final Map<String, Object> topicPersistLocks = new ConcurrentHashMap<>();
+
+ // Runtime random-write index. The on-disk .meta file is the only source of truth; this map is
+ // rebuilt from .meta during recovery and is never persisted separately.
+ private final Map<String, Map<String, ProgressIndexEntry>> topicProgressIndexes =
+ new ConcurrentHashMap<>();
+
Review Comment:
Done in 5ea428025e. Added comments explaining commitStateKeys and the outer/inner keys of topicProgressIndexes.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -238,42 +225,136 @@ public List<SubscriptionCommitContext> commit(
return allSuccessful;
}
- public void seek(
- final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
- final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ public int refreshInFlightEventLeases(
+ final ConsumerConfig consumerConfig,
+ final List<SubscriptionCommitContext> processorBufferedCommitContexts) {
+ if (Objects.isNull(processorBufferedCommitContexts)
+ || processorBufferedCommitContexts.isEmpty()) {
+ return 0;
+ }
- final ConsensusSubscriptionBroker consensusBroker =
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+ final String consumerId = consumerConfig.getConsumerId();
+ final ISubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ISubscriptionBroker consensusBroker =
consumerGroupIdToConsensusBroker.get(consumerGroupId);
Review Comment:
Done in aa6810e062. SubscriptionBrokerAgent now stores brokers in one consumerGroupId -> broker list map instead of separate pipe/consensus maps.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -1129,7 +1125,40 @@ public SubscriptionEvent pollTablets(
"ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
this, consumerId, commitContext));
}
- return event;
+ if (Objects.isNull(event.getCurrentResponse())
+ || event.getCurrentResponse().getResponseType()
+ != SubscriptionPollResponseType.TABLETS.getType()
+ || !(event.getCurrentResponse().getPayload() instanceof TabletsPayload)) {
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: unexpected in-flight response for consumer %s, "
+ + "commit context %s, offset %s",
Review Comment:
Done in 5ea428025e. The unexpected in-flight consensus response message now uses a DataNodeMiscMessages i18n constant.
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/service/IoTConsensusRPCServiceProcessor.java:
##########
Review Comment:
Done in 5ea428025e. The hard-coded consensus group message now uses the existing i18n message constant.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -397,22 +473,32 @@ private String buildUnsupportedConsensusRuntimeMessage(
}
public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
- final String consumerGroupId = commitContext.getConsumerGroupId();
- final String topicName = commitContext.getTopicName();
+ final ISubscriptionBroker broker =
+ getBrokerForTopic(commitContext.getConsumerGroupId(), commitContext.getTopicName());
+ return Objects.isNull(broker) || broker.isCommitContextOutdated(commitContext);
+ }
- // Try consensus broker first
+ private ISubscriptionBroker getBrokerForTopic(
+ final String consumerGroupId, final String topicName) {
final ConsensusSubscriptionBroker consensusBroker =
consumerGroupIdToConsensusBroker.get(consumerGroupId);
if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
- return consensusBroker.isCommitContextOutdated(commitContext);
+ return consensusBroker;
}
+ return consumerGroupIdToPipeBroker.get(consumerGroupId);
+ }
- // Fall back to pipe broker
- final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
- if (Objects.isNull(pipeBroker)) {
- return true;
+ private ISubscriptionBroker getBrokerForTopicOrThrow(
+ final String consumerGroupId, final String topicName) {
+ final ISubscriptionBroker broker = getBrokerForTopic(consumerGroupId, topicName);
+ if (Objects.nonNull(broker)) {
+ return broker;
}
- return pipeBroker.isCommitContextOutdated(commitContext);
+ final String errorMessage =
+ String.format(
+ "Subscription: broker bound to consumer group [%s] does not exist", consumerGroupId);
Review Comment:
Done in 5ea428025e. The broker messages in this area were moved to DataNodeMiscMessages with both en/zh entries.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusSubscriptionCommitManager.java:
##########
@@ -529,33 +570,210 @@ public void receiveProgressBroadcast(
// ======================== Helper Methods ========================
- // Use a separator that cannot appear in consumerGroupId, topicName, or regionId
- // to prevent key collisions (e.g., "a_b" + "c" vs "a" + "b_c").
+ // Kept as the in-memory and ConfigNode sync key separator for the existing progress protocol.
private static final String KEY_SEPARATOR = "##";
private String generateKey(
final String consumerGroupId, final String topicName, final ConsensusGroupId regionId) {
- return consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionId.toString();
+ return generateKey(consumerGroupId, topicName, regionId.toString());
}
- private File getProgressFile(final String key) {
- return new File(persistDir, PROGRESS_FILE_PREFIX + key + PROGRESS_FILE_SUFFIX);
+ private String generateKey(
+ final String consumerGroupId, final String topicName, final String regionIdStr) {
+ return consumerGroupId + KEY_SEPARATOR + topicName + KEY_SEPARATOR + regionIdStr;
}
- private ConsensusSubscriptionCommitState tryRecover(final String key, final String regionIdStr) {
- final File file = getProgressFile(key);
- if (!file.exists()) {
+ private CommitStateKey getCommitStateKey(
+ final String consumerGroupId, final String topicName, final String regionIdStr) {
+ final String stateKey = generateKey(consumerGroupId, topicName, regionIdStr);
+ return commitStateKeys.computeIfAbsent(
+ stateKey,
+ ignored ->
+ new CommitStateKey(
+ consumerGroupId,
+ topicName,
+ regionIdStr,
+ stateKey,
+ generateTopicFileKey(consumerGroupId, topicName)));
+ }
+
+ private String generateTopicFileKey(final String consumerGroupId, final String topicName) {
+ return encodeFileNameComponent(consumerGroupId)
+ + KEY_SEPARATOR
+ + encodeFileNameComponent(topicName);
+ }
+
+ private static String encodeFileNameComponent(final String value) {
+ return Base64.getUrlEncoder()
+ .withoutPadding()
+ .encodeToString(String.valueOf(value).getBytes(StandardCharsets.UTF_8));
+ }
+
+ private static String decodeFileNameComponent(final String value) {
+ switch (value.length() & 3) {
+ case 0:
+ return new String(Base64.getUrlDecoder().decode(value), StandardCharsets.UTF_8);
+ case 2:
+ return new String(Base64.getUrlDecoder().decode(value + "=="), StandardCharsets.UTF_8);
+ case 3:
+ return new String(Base64.getUrlDecoder().decode(value + "="), StandardCharsets.UTF_8);
+ default:
+ throw new IllegalArgumentException("Invalid base64 url component length");
+ }
+ }
+
+ private static String[] decodeTopicFileKey(final String topicFileKey) {
+ final int separatorIndex = topicFileKey.indexOf(KEY_SEPARATOR);
+ if (separatorIndex < 0) {
return null;
}
- try (final FileInputStream fis = new FileInputStream(file)) {
- final byte[] bytes = new byte[(int) file.length()];
- fis.read(bytes);
- final ByteBuffer buffer = ByteBuffer.wrap(bytes);
- return ConsensusSubscriptionCommitState.deserialize(regionIdStr, buffer);
- } catch (final IOException e) {
- LOGGER.warn("Failed to recover consensus subscription progress from {}", file, e);
+ try {
+ return new String[] {
+ decodeFileNameComponent(topicFileKey.substring(0, separatorIndex)),
+ decodeFileNameComponent(topicFileKey.substring(separatorIndex + KEY_SEPARATOR.length()))
+ };
+ } catch (final IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ private Object getTopicPersistLock(final String topicFileKey) {
+ return topicPersistLocks.computeIfAbsent(topicFileKey, ignored -> new Object());
+ }
+
+ private File getMetaFile(final String topicFileKey) {
+ return new File(persistDir, PROGRESS_FILE_PREFIX + topicFileKey + PROGRESS_META_FILE_SUFFIX);
+ }
+
+ private File getLegacyIndexFile(final String topicFileKey) {
+ return new File(
+ persistDir, PROGRESS_FILE_PREFIX + topicFileKey + LEGACY_PROGRESS_INDEX_FILE_SUFFIX);
+ }
+
+ private void recoverAllTopicStatesIfNeeded() {
+ final File dir = new File(persistDir);
+ final File[] metaFiles =
+ dir.listFiles(
+ (ignored, name) ->
+ name.startsWith(PROGRESS_FILE_PREFIX) && name.endsWith(PROGRESS_META_FILE_SUFFIX));
+ if (Objects.isNull(metaFiles)) {
+ return;
+ }
+ for (final File metaFile : metaFiles) {
+ final String topicFileKey = extractTopicFileKey(metaFile.getName());
+ if (Objects.isNull(topicFileKey) || recoveredTopicKeys.contains(topicFileKey)) {
+ continue;
+ }
+ final String[] decodedTopicKey = decodeTopicFileKey(topicFileKey);
+ if (Objects.isNull(decodedTopicKey)) {
+ LOGGER.warn(
+ "Skip malformed consensus subscription progress file name {}", metaFile.getName());
+ continue;
+ }
+ recoverTopicStatesIfNeeded(topicFileKey, decodedTopicKey[0], decodedTopicKey[1]);
+ }
+ }
+
+ private String extractTopicFileKey(final String metaFileName) {
+ if (!metaFileName.startsWith(PROGRESS_FILE_PREFIX)
+ || !metaFileName.endsWith(PROGRESS_META_FILE_SUFFIX)) {
return null;
}
+ return metaFileName.substring(
+ PROGRESS_FILE_PREFIX.length(), metaFileName.length() - PROGRESS_META_FILE_SUFFIX.length());
+ }
+
+ private void recoverTopicStatesIfNeeded(final CommitStateKey stateKey) {
+ recoverTopicStatesIfNeeded(stateKey.topicFileKey, stateKey.consumerGroupId, stateKey.topicName);
+ }
+
+ private void recoverTopicStatesIfNeeded(
+ final String topicFileKey, final String consumerGroupId, final String topicName) {
+ if (recoveredTopicKeys.contains(topicFileKey)) {
+ return;
+ }
+ synchronized (getTopicPersistLock(topicFileKey)) {
+ if (recoveredTopicKeys.contains(topicFileKey)) {
+ return;
+ }
+ try {
+ final TopicProgressSnapshot snapshot = readTopicProgressSnapshot(topicFileKey);
+ for (final Map.Entry<String, ConsensusSubscriptionCommitState> entry :
+ snapshot.states.entrySet()) {
+ final CommitStateKey recoveredStateKey =
+ getCommitStateKey(consumerGroupId, topicName, entry.getKey());
+ commitStates.putIfAbsent(recoveredStateKey.stateKey, entry.getValue());
+ }
+ topicProgressIndexes.put(topicFileKey, snapshot.indexEntries);
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "Failed to recover consensus subscription progress for consumerGroupId={}, "
+ + "topicName={}",
+ consumerGroupId,
+ topicName,
+ e);
+ topicProgressIndexes.put(topicFileKey, Collections.emptyMap());
+ } finally {
+ recoveredTopicKeys.add(topicFileKey);
+ }
+ }
+ }
+
+ // Recovery reads .meta sequentially and rebuilds the runtime offset index.
+ private TopicProgressSnapshot readTopicProgressSnapshot(final String topicFileKey)
+ throws IOException {
+ final File metaFile = getMetaFile(topicFileKey);
+ if (!metaFile.exists()) {
+ return TopicProgressSnapshot.empty();
+ }
+ try {
+ final ByteBuffer buffer = ByteBuffer.wrap(Files.readAllBytes(metaFile.toPath()));
+ final int version = ReadWriteIOUtils.readInt(buffer);
+ if (version != TOPIC_PROGRESS_FILE_VERSION) {
+ throw new IOException(
+ "Unsupported consensus subscription progress file version " + version);
Review Comment:
Done in 5ea428025e. The unsupported progress file version message now uses DataNodeMiscMessages en/zh constants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3109399421
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/WatermarkProcessor.java:
##########
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.subscription.consumer.base;
+
+import org.apache.iotdb.session.subscription.payload.SubscriptionMessage;
+import org.apache.iotdb.session.subscription.payload.SubscriptionMessageType;
+
+import org.apache.tsfile.write.record.Tablet;
+
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.PriorityQueue;
+
+/**
+ * A buffering processor that reorders messages based on watermark semantics. Messages are buffered
+ * internally and emitted only when the watermark advances past their maximum timestamp.
+ *
+ * <p>Watermark = (minimum of latest timestamp per <b>active</b> source) - maxOutOfOrdernessMs
+ *
+ * <p>A source is considered "stale" if its latest timestamp has not increased for {@code
+ * staleSourceTimeoutMs}. Stale sources are excluded from the watermark calculation, preventing a
+ * single slow or idle source from anchoring the global watermark indefinitely.
+ *
+ * <p>Server-side WATERMARK events (carrying per-region timestamp progress) serve as heartbeats,
+ * confirming source liveness. They advance the per-source timestamp only when their timestamp is
+ * higher than the previously observed value.
+ *
+ * <p>A timeout mechanism ensures that buffered messages are eventually flushed even if no new data
+ * arrives, preventing unbounded buffering.
+ *
+ * <p><b>Note:</b> This processor is primarily intended as a reference implementation. For
+ * production use with large-scale out-of-order data, consider using a downstream stream processing
+ * framework (Flink, Spark) for watermark handling.
+ */
+public class WatermarkProcessor implements SubscriptionMessageProcessor {
+
+ private static final long DEFAULT_STALE_SOURCE_TIMEOUT_MS = 30_000L;
+ private static final long DEFAULT_MAX_BUFFER_BYTES = 64L * 1024 * 1024; // 64 MB
+
+ private final long maxOutOfOrdernessMs;
+ private final long timeoutMs;
+ private final long staleSourceTimeoutMs;
+ private final long maxBufferBytes;
+
+ // Buffer ordered by message max timestamp
+ private final PriorityQueue<TimestampedMessage> buffer =
+ new PriorityQueue<>((a, b) -> Long.compare(a.maxTimestamp, b.maxTimestamp));
+
+ // Track latest timestamp per source (deviceId/tableName)
+ private final java.util.Map<String, Long> latestPerSource = new java.util.HashMap<>();
+ // Track wall-clock time when each source's timestamp last increased
+ private final java.util.Map<String, Long> lastAdvancedTimeMs = new java.util.HashMap<>();
+ private long lastEmitTimeMs = System.currentTimeMillis();
+ private long bufferedBytes = 0;
+
+ // Current watermark value
+ private long watermark = Long.MIN_VALUE;
+
+ /**
+ * Creates a WatermarkProcessor with default stale source timeout (30 seconds).
+ *
+ * @param maxOutOfOrdernessMs maximum expected out-of-orderness in milliseconds
+ * @param timeoutMs if no data arrives within this duration, force-flush all buffered messages
+ */
+ public WatermarkProcessor(final long maxOutOfOrdernessMs, final long timeoutMs) {
+ this(maxOutOfOrdernessMs, timeoutMs, DEFAULT_STALE_SOURCE_TIMEOUT_MS, DEFAULT_MAX_BUFFER_BYTES);
+ }
+
+ /**
+ * Creates a WatermarkProcessor.
+ *
+ * @param maxOutOfOrdernessMs maximum expected out-of-orderness in milliseconds
+ * @param timeoutMs if no data arrives within this duration, force-flush all buffered messages
+ * @param staleSourceTimeoutMs if a source's timestamp has not increased for this duration, it is
+ * excluded from watermark calculation. Use {@link Long#MAX_VALUE} to disable.
+ * @param maxBufferBytes maximum total estimated bytes of buffered messages. When exceeded, all
+ * buffered messages are force-flushed regardless of watermark. Defaults to 64 MB.
+ */
+ public WatermarkProcessor(
+ final long maxOutOfOrdernessMs,
+ final long timeoutMs,
+ final long staleSourceTimeoutMs,
+ final long maxBufferBytes) {
+ this.maxOutOfOrdernessMs = maxOutOfOrdernessMs;
+ this.timeoutMs = timeoutMs;
+ this.staleSourceTimeoutMs = staleSourceTimeoutMs;
+ this.maxBufferBytes = maxBufferBytes;
+ }
+
+ @Override
+ public List<SubscriptionMessage> process(final List<SubscriptionMessage> messages) {
+ final long now = System.currentTimeMillis();
+
+ // Buffer incoming messages and update per-source timestamps
+ for (final SubscriptionMessage message : messages) {
+ // WATERMARK events carry server-side timestamp progress per region.
+ // They serve as heartbeats and advance per-source tracking only when the timestamp
+ // actually increases.
+ if (message.getMessageType() == SubscriptionMessageType.WATERMARK.getType()) {
+ final String regionKey =
+ "region-"
+ + message.getCommitContext().getDataNodeId()
+ + "-"
+ + message.getCommitContext().getRegionId();
+ advanceSourceTimestamp(regionKey, message.getWatermarkTimestamp(), now);
+ continue; // Do not buffer system events
+ }
+
+ final long maxTs = extractMaxTimestamp(message);
+ final long estimatedSize = message.estimateSize();
+ buffer.add(new TimestampedMessage(message, maxTs, estimatedSize));
+ bufferedBytes += estimatedSize;
+ updateSourceTimestamp(message, maxTs, now);
+ }
+
+ // Compute watermark = min(latest per active source) - maxOutOfOrderness
+ // Sources whose timestamp has not increased for staleSourceTimeoutMs are excluded.
+ if (!latestPerSource.isEmpty()) {
+ long minLatest = Long.MAX_VALUE;
+ for (final java.util.Map.Entry<String, Long> entry : latestPerSource.entrySet()) {
+ final Long lastAdv = lastAdvancedTimeMs.get(entry.getKey());
+ if (lastAdv != null && (now - lastAdv) <= staleSourceTimeoutMs) {
+ minLatest = Math.min(minLatest, entry.getValue());
+ }
+ }
+ if (minLatest != Long.MAX_VALUE) {
+ watermark = minLatest - maxOutOfOrdernessMs;
+ }
+ // If all sources are stale, watermark stays unchanged and timeout will handle it.
+ }
+
+ // Emit messages whose maxTimestamp <= watermark
+ final List<SubscriptionMessage> emitted = emit(watermark);
+
+ // Buffer overflow: force-flush all if buffer exceeds byte limit
+ if (bufferedBytes > maxBufferBytes) {
+ return forceFlushAll();
Review Comment:
When `bufferedBytes > maxBufferBytes`, this branch returns `forceFlushAll()` and discards the `emitted` list that was already produced on line 151. Those messages have been polled out of the buffer (and `bufferedBytes` already decremented) inside `emit()`, so they are not in the buffer anymore and never reach the caller — they are silently dropped.
Suggest merging `emitted` with the force-flush result before returning, e.g.:
```java
if (bufferedBytes > maxBufferBytes) {
final List<SubscriptionMessage> rest = forceFlushAll();
emitted.addAll(rest);
return emitted;
}
```
(Same shape applies to the timeout branch below if you ever hit both paths in one call, though in the current code `emitted.isEmpty()` guards that one.)
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/ColumnAlignProcessor.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.subscription.consumer.base;
+
+import org.apache.iotdb.session.subscription.payload.SubscriptionMessage;
+import org.apache.iotdb.session.subscription.payload.SubscriptionMessageType;
+
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * A non-buffering processor that forward-fills null columns in each Tablet using the last known
+ * value for the same device/table. This is useful for CDC scenarios where a write only updates a
+ * subset of columns, leaving others null; the processor fills them with the most recent value.
+ *
+ * <p>State is maintained per device (identified by {@code Tablet.getDeviceId()} for tree-model or
+ * {@code Tablet.getTableName()} for table-model).
+ */
+public class ColumnAlignProcessor implements SubscriptionMessageProcessor {
+
+ // deviceKey -> (columnIndex -> lastValue)
+ private final Map<String, Map<Integer, Object>> lastValues = new HashMap<>();
Review Comment:
The per-device cache is keyed by column **index** (`col`), not column name. If the same device / table appears in a later batch with a different column ordering — e.g. a dropped / added column in the table model, or any schema change between WAL batches — the forward-fill at line 81-84 will pull the previous column's cached value into the new column sitting at the same index, silently corrupting data.
Suggest keying on column name (`tablet.getSchemas().get(col).getMeasurementId()` or the equivalent table-model accessor) so the cache survives reorderings and column add/drop. Even if the current producer always sends a stable ordering per device, an index-based cache is a fragile contract to rely on across independent batches — any future producer-side change (column pruning, projection, schema evolution) would corrupt data without any visible error.
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/WriterMeta.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.consensus.iot;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.nio.file.StandardCopyOption;
+import java.nio.file.StandardOpenOption;
+import java.util.Optional;
+
+final class WriterMeta {
+
+ private static final int FORMAT_VERSION = 1;
+
+ private final long writerEpoch;
+ private final long lastAllocatedLocalSeq;
+ private final long lastAssignedPhysicalTimeMs;
+
+ WriterMeta(long writerEpoch, long lastAllocatedLocalSeq, long lastAssignedPhysicalTimeMs) {
+ this.writerEpoch = writerEpoch;
+ this.lastAllocatedLocalSeq = lastAllocatedLocalSeq;
+ this.lastAssignedPhysicalTimeMs = lastAssignedPhysicalTimeMs;
+ }
+
+ long getWriterEpoch() {
+ return writerEpoch;
+ }
+
+ long getLastAllocatedLocalSeq() {
+ return lastAllocatedLocalSeq;
+ }
+
+ long getLastAssignedPhysicalTimeMs() {
+ return lastAssignedPhysicalTimeMs;
+ }
+
+ static Optional<WriterMeta> load(Path path) throws IOException {
+ if (!Files.exists(path)) {
+ return Optional.empty();
+ }
+ try (InputStream inputStream = Files.newInputStream(path, StandardOpenOption.READ);
+ DataInputStream dataInputStream = new DataInputStream(inputStream)) {
+ final int version = dataInputStream.readInt();
+ if (version != FORMAT_VERSION) {
+ throw new IOException(
+ String.format(
+ "Unsupported writer meta version %d in %s", version, path.toAbsolutePath()));
+ }
+ return Optional.of(
+ new WriterMeta(
+ dataInputStream.readLong(), dataInputStream.readLong(), dataInputStream.readLong()));
+ }
+ }
+
+ void persist(Path path) throws IOException {
+ final Path parent = path.getParent();
+ if (parent != null && !Files.exists(parent)) {
+ Files.createDirectories(parent);
+ }
+ final Path tempPath =
+ parent == null
+ ? Paths.get(path + ".tmp")
+ : parent.resolve(path.getFileName().toString() + ".tmp");
+ try (OutputStream outputStream =
+ Files.newOutputStream(
+ tempPath,
+ StandardOpenOption.CREATE,
+ StandardOpenOption.TRUNCATE_EXISTING,
+ StandardOpenOption.WRITE);
+ DataOutputStream dataOutputStream = new DataOutputStream(outputStream)) {
+ dataOutputStream.writeInt(FORMAT_VERSION);
+ dataOutputStream.writeLong(writerEpoch);
+ dataOutputStream.writeLong(lastAllocatedLocalSeq);
+ dataOutputStream.writeLong(lastAssignedPhysicalTimeMs);
+ dataOutputStream.flush();
+ }
+ Files.move(tempPath, path, StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
Review Comment:
`DataOutputStream.flush()` on line 98 only pushes bytes from the JVM buffer into the OS page cache — it does not guarantee the temp file content is on disk before the atomic rename on line 100. On power loss between the close and a later (implicit) fsync, the rename can succeed while the temp file's data is still 0 bytes or truncated. On recovery that shows up as either `readInt(version) != FORMAT_VERSION` (throws, blocks startup) or silently corrupt `writerEpoch` / `lastAllocatedLocalSeq`, which are the authoritative inputs for writer-epoch / localSeq recovery.
Suggest fsync'ing the temp file before the move — e.g. obtain a `FileChannel` on the temp path and call `channel.force(true)` after the writes, and ideally fsync the parent directory after the rename as well. This matches the pattern used by other metadata-file persistence paths in the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279122167
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
+ }
+
+ private static final class SearchableRequestMeta {
+ private final long searchIndex;
+ private final long physicalTime;
+ private final int nodeId;
+ private final long localSeq;
+
+ private SearchableRequestMeta(
+ final long searchIndex, final long physicalTime, final int nodeId, final long localSeq) {
+ this.searchIndex = searchIndex;
+ this.physicalTime = physicalTime;
+ this.nodeId = nodeId;
+ this.localSeq = localSeq;
+ }
+ }
+
+ private static void forEachSealedSearchableRequest(
+ final File logDir, final SearchableRequestVisitor visitor) {
+ final File[] walFiles = listSealedWALFiles(logDir);
+ if (walFiles == null || walFiles.length == 0) {
+ return;
+ }
+
+ for (final File walFile : walFiles) {
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ long pendingSearchIndex = Long.MIN_VALUE;
+ long pendingPhysicalTime = 0L;
+ int pendingNodeId = -1;
+ long pendingLocalSeq = Long.MIN_VALUE;
+ boolean hasPending = false;
+
+ while (reader.hasNext()) {
+ final ByteBuffer buffer = reader.next();
+ final WALEntryType type = WALEntryType.valueOf(buffer.get());
+ buffer.clear();
+ if (!type.needSearch()) {
+ continue;
+ }
+
+ final long currentLocalSeq = reader.getCurrentEntryLocalSeq();
+ final long currentPhysicalTime = reader.getCurrentEntryPhysicalTime();
+ final int currentNodeId = reader.getCurrentEntryNodeId();
+
+ buffer.position(SEARCH_INDEX_OFFSET);
+ final long bodySearchIndex = buffer.getLong();
+ buffer.clear();
+ final long currentSearchIndex = bodySearchIndex >= 0 ? bodySearchIndex : currentLocalSeq;
+
+ if (hasPending
+ && pendingPhysicalTime == currentPhysicalTime
+ && pendingNodeId == currentNodeId
+ && pendingLocalSeq == currentLocalSeq) {
+ if (pendingSearchIndex < 0 && currentSearchIndex >= 0) {
+ pendingSearchIndex = currentSearchIndex;
+ }
+ continue;
+ }
+
+ if (hasPending
+ && !visitor.onRequest(
+ new SearchableRequestMeta(
+ pendingSearchIndex >= 0 ? pendingSearchIndex : pendingLocalSeq,
+ pendingPhysicalTime,
+ pendingNodeId,
+ pendingLocalSeq))) {
+ return;
+ }
+
+ hasPending = true;
+ pendingSearchIndex = currentSearchIndex;
+ pendingPhysicalTime = currentPhysicalTime;
+ pendingNodeId = currentNodeId;
+ pendingLocalSeq = currentLocalSeq;
+ }
+
+ if (hasPending
+ && !visitor.onRequest(
+ new SearchableRequestMeta(
+ pendingSearchIndex >= 0 ? pendingSearchIndex : pendingLocalSeq,
+ pendingPhysicalTime,
+ pendingNodeId,
+ pendingLocalSeq))) {
+ return;
+ }
+ } catch (final IOException e) {
+ logger.warn("Failed to scan WAL file {} for searchable request metadata", walFile, e);
+ }
+ }
+ }
+
+ private static int compareCompatibleProgress(
+ final long leftPhysicalTime,
+ final int leftNodeId,
+ final long leftLocalSeq,
+ final long rightPhysicalTime,
+ final long rightLocalSeq) {
+ if (leftPhysicalTime != rightPhysicalTime) {
+ return Long.compare(leftPhysicalTime, rightPhysicalTime);
+ }
+ if (leftLocalSeq != rightLocalSeq) {
+ return Long.compare(leftLocalSeq, rightLocalSeq);
+ }
+ return 0;
+ }
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279153573
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
Review Comment:
Replaced the Pair key with a dedicated InFlightEventKey containing consumerId and commitContext.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279152193
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
Review Comment:
Added a comment explaining how seekGeneration distinguishes events before and after seek.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279168401
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = inFlightEvents.get(new Pair<>(consumerId, commitContext));
+ if (Objects.isNull(event)) {
+ if (isCommitContextOutdated(commitContext)) {
+ return generateOutdatedErrorResponse();
+ }
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
+ this, consumerId, commitContext));
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ // ======================== Prefetch Round Drive ========================
+
+ private static final long WAL_GAP_RETRY_SLEEP_MS = 10L;
+ private static final long WAL_GAP_WAIT_LOG_INTERVAL_MS = 5_000L;
+
+ private static final long PREFETCH_STATS_LOG_INTERVAL_MS = 5_000L;
+
+ public PrefetchRoundResult drivePrefetchOnce() {
+ if (applyPendingSeekRequestIfNecessary()) {
+ return closeRequested ? PrefetchRoundResult.dormant() : PrefetchRoundResult.rescheduleNow();
+ }
+
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !prefetchInitialized) {
+ return PrefetchRoundResult.dormant();
+ }
+
+ logPeriodicStatsIfNecessary();
+
+ final long currentSeekGeneration = seekGeneration.get();
+ if (currentSeekGeneration != observedSeekGeneration) {
+ resetRoundStateForSeek(currentSeekGeneration);
+ }
+
+ applyPendingSubscriptionWalReset(observedSeekGeneration);
+ recycleInFlightEvents();
+
+ if (!isActive || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return computeIdleRoundResult();
+ }
+
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+ final int batchMaxDelayMs = config.getSubscriptionConsensusBatchMaxDelayInMs();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+
+ final List<IndexedConsensusRequest> batch = drainPendingEntries(maxWalEntries);
+ if (!batch.isEmpty()) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: drained {} entries from pendingEntries, "
+ + "first searchIndex={}, last searchIndex={}, nextExpected={}, "
+ + "prefetchingQueueSize={}",
+ this,
+ batch.size(),
+ batch.get(0).getSearchIndex(),
+ batch.get(batch.size() - 1).getSearchIndex(),
+ nextExpectedSearchIndex.get(),
+ prefetchingQueue.size());
+
+ final boolean batchAccepted =
+ accumulateFromPending(
+ batch, lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes);
+ if (!batchAccepted) {
+ if (pendingWalGapRetryRequested) {
+ // Once a drained batch hits an unresolved WAL gap, the affected suffix falls back to
+ // the WAL path on later rounds instead of being requeued into the bounded pending path.
+ return PrefetchRoundResult.rescheduleAfter(WAL_GAP_RETRY_SLEEP_MS);
+ }
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ }
+
+ if (batch.isEmpty() && lingerBatch.isEmpty()) {
+ tryCatchUpFromWAL(observedSeekGeneration);
+ }
+
+ if (!drainBufferedRealtimeLanes(
+ lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes)) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerElapsedMs = System.currentTimeMillis() - lingerBatch.firstTabletTimeMs;
+ if (lingerElapsedMs >= batchMaxDelayMs) {
+ if (seekGeneration.get() != observedSeekGeneration) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: time-based flush, {} tablets lingered for {}ms "
+ + "(threshold={}ms)",
+ this,
+ lingerBatch.tablets.size(),
+ lingerElapsedMs,
+ batchMaxDelayMs);
+ flushBatch(lingerBatch, observedSeekGeneration);
+ }
+ }
+
+ maybeInjectWatermark();
+ return computeIdleRoundResult();
+ } catch (final Throwable fatal) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: prefetch round failed " + "(type={}, message={})",
+ this,
+ fatal.getClass().getName(),
+ fatal.getMessage(),
+ fatal);
+ if (fatal instanceof VirtualMachineError) {
+ markClosed();
+ return PrefetchRoundResult.dormant();
+ }
+ return PrefetchRoundResult.rescheduleAfter(100L);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private void logPeriodicStatsIfNecessary() {
+ final long nowMs = System.currentTimeMillis();
+ if (nowMs - lastStatsLogTimeMs < PREFETCH_STATS_LOG_INTERVAL_MS) {
+ return;
+ }
+
+ final long currentPendingAcceptedEntries = pendingPathAcceptedEntries.get();
+ final long currentWalAcceptedEntries = walPathAcceptedEntries.get();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: periodic stats, lag={}, pendingDelta={}, walDelta={}, "
+ + "pendingTotal={}, walTotal={}, pendingQueueSize={}, prefetchingQueueSize={}, "
+ + "inFlightEventsSize={}, realtimeLaneCount={}, walHasNext={}, isActive={}, subtaskScheduled={}",
+ this,
+ getLag(),
+ currentPendingAcceptedEntries - lastPendingAcceptedEntries,
+ currentWalAcceptedEntries - lastWalAcceptedEntries,
+ currentPendingAcceptedEntries,
+ currentWalAcceptedEntries,
+ pendingEntries.size(),
+ prefetchingQueue.size(),
+ inFlightEvents.size(),
+ realtimeEntriesByLane.size(),
+ hasReadableWalEntries(),
+ isActive,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ lastStatsLogTimeMs = nowMs;
+ lastPendingAcceptedEntries = currentPendingAcceptedEntries;
+ lastWalAcceptedEntries = currentWalAcceptedEntries;
+ }
+
+ private void resetRoundStateForSeek(final long newSeekGeneration) {
+ restorePendingSubscriptionWalCursor(newSeekGeneration);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = newSeekGeneration;
+ }
+
+ private List<IndexedConsensusRequest> drainPendingEntries(final int maxWalEntries) {
+ final List<IndexedConsensusRequest> batch = new ArrayList<>();
+ IndexedConsensusRequest next;
+ while (batch.size() < maxWalEntries && (next = pendingEntries.poll()) != null) {
+ batch.add(next);
+ }
+ return batch;
+ }
+
+ private PrefetchRoundResult computeIdleRoundResult() {
+ if (isClosed || !prefetchInitialized || !isActive) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (hasImmediatePrefetchableWork()) {
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ long delayMs = Long.MAX_VALUE;
+ if (hasHistoricalWalLag()) {
+ delayMs = Math.min(delayMs, WAL_GAP_RETRY_SLEEP_MS);
+ }
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerDelayMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusBatchMaxDelayInMs()
+ - (System.currentTimeMillis() - lingerBatch.firstTabletTimeMs);
+ delayMs = Math.min(delayMs, Math.max(1L, lingerDelayMs));
+ }
+
+ final long watermarkDelayMs = computeWatermarkDelayMs();
+ if (watermarkDelayMs > 0L) {
+ delayMs = Math.min(delayMs, watermarkDelayMs);
+ }
+
+ if (!inFlightEvents.isEmpty()) {
+ delayMs =
+ Math.min(
+ delayMs,
+ SubscriptionConfig.getInstance().getSubscriptionRecycleUncommittedEventIntervalMs());
+ }
+
+ return delayMs == Long.MAX_VALUE
+ ? PrefetchRoundResult.dormant()
+ : PrefetchRoundResult.rescheduleAfter(delayMs);
+ }
+
+ private long computeWatermarkDelayMs() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return -1L;
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0L) {
+ return -1L;
+ }
+ if (lastWatermarkEmitTimeMs == 0L) {
+ return 1L;
+ }
+ final long elapsedMs = System.currentTimeMillis() - lastWatermarkEmitTimeMs;
+ return elapsedMs >= intervalMs ? 1L : Math.max(1L, intervalMs - elapsedMs);
+ }
+
+ private boolean hasImmediatePrefetchableWork() {
+ return !pendingEntries.isEmpty() || !realtimeEntriesByLane.isEmpty() || hasReadableWalEntries();
+ }
+
+ private boolean hasHistoricalWalLag() {
+ return nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex();
+ }
+
+ /**
+ * Accumulates tablets from pending entries into the linger buffer. When pending replay outruns
+ * the local WAL reader, this method backfills the local-index gap from WAL before continuing.
+ *
+ * @return false if the batch became stale because seek generation changed while flushing
+ */
+ private static boolean hasLocalSearchIndex(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() >= 0;
+ }
+
+ private boolean isBeforeLocalCursor(final IndexedConsensusRequest request) {
+ return hasLocalSearchIndex(request) && request.getSearchIndex() < nextExpectedSearchIndex.get();
+ }
+
+ private void advanceLocalCursorIfPresent(final IndexedConsensusRequest request) {
+ if (hasLocalSearchIndex(request)) {
+ nextExpectedSearchIndex.set(request.getSearchIndex() + 1);
+ }
+ }
+
+ private boolean appendRealtimeRequest(
+ final IndexedConsensusRequest request,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean fromPending) {
+ final PreparedEntry preparedEntry = prepareEntry(request);
+ if (Objects.isNull(preparedEntry)) {
+ return true;
+ }
+ if (!appendPreparedEntryViaRealtimeLane(
+ batchState, preparedEntry, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+ if (fromPending) {
+ markAcceptedFromPending();
+ } else {
+ markAcceptedFromWal();
+ }
+ return true;
+ }
+
+ private boolean accumulateFromPending(
+ final List<IndexedConsensusRequest> batch,
+ final DeliveryBatchState lingerBatch,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+
+ int processedCount = 0;
+ int skippedCount = 0;
+
+ for (int index = 0; index < batch.size(); index++) {
+ final IndexedConsensusRequest request = batch.get(index);
+ final long searchIndex = request.getSearchIndex();
+
+ // Only local-indexed requests participate in the internal WAL read cursor.
+ final long expected = nextExpectedSearchIndex.get();
+ if (hasLocalSearchIndex(request) && searchIndex > expected) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: gap detected, expected={}, got={}. "
+ + "Filling {} entries from WAL.",
+ this,
+ expected,
+ searchIndex,
+ searchIndex - expected);
+ if (!fillGapFromWAL(
+ expected,
+ searchIndex,
+ lingerBatch,
+ expectedSeekGeneration,
+ maxTablets,
+ maxBatchBytes)) {
+ return false;
+ }
+ }
+
+ if (isBeforeLocalCursor(request)) {
+ skippedCount++;
+ continue;
+ }
+
+ if (shouldSkipForRecoveryProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ request, lingerBatch, expectedSeekGeneration, maxTablets, maxBatchBytes, true)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(request);
+ processedCount++;
+ advanceLocalCursorIfPresent(request);
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: accumulate complete, batchSize={}, processed={}, "
+ + "skipped={}, lingerTablets={}, nextExpected={}",
+ this,
+ batch.size(),
+ processedCount,
+ skippedCount,
+ lingerBatch.tablets.size(),
+ nextExpectedSearchIndex.get());
+
+ return true;
+ }
+
+ /**
+ * Fills a gap in the pending queue by reading entries from WAL so the internal local replay
+ * cursor stays contiguous even when pending delivery jumps ahead of the WAL iterator.
+ *
+ * <p>Temporary WAL visibility lag is treated as a normal back-pressure condition: once a drained
+ * pending batch encounters an unresolved local-index gap, the queue backs off and lets the
+ * affected suffix fall back to the WAL path on later rounds. This keeps replay contiguous without
+ * requeueing the drained batch back into the bounded pending queue.
+ *
+ * @return false if gap fill had to stop because the current batch became stale or the queue was
+ * interrupted/closed
+ */
+ private boolean fillGapFromWAL(
+ final long fromIndex,
+ final long toIndex,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ pendingWalGapRetryRequested = false;
+ resetSubscriptionWALPosition(fromIndex);
+ if (seekGeneration.get() != expectedSeekGeneration || isClosed) {
+ return false;
+ }
+ if (!pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, Integer.MAX_VALUE, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final long nextExpected = nextExpectedSearchIndex.get();
+ if (nextExpected >= toIndex) {
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ return true;
+ }
+
+ final long nowMs = System.currentTimeMillis();
+ if (walGapWaitStartTimeMs == 0L) {
+ walGapWaitStartTimeMs = nowMs;
+ }
+ if (lastWalGapWaitLogTimeMs == 0L
+ || nowMs - lastWalGapWaitLogTimeMs >= WAL_GAP_WAIT_LOG_INTERVAL_MS) {
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: waiting {}ms for WAL gap [{}, {}) to become visible, "
+ + "currentNextExpected={}, currentWalIndex={}, seekGeneration={}",
+ this,
+ nowMs - walGapWaitStartTimeMs,
+ nextExpected,
+ toIndex,
+ nextExpected,
+ consensusReqReader.getCurrentSearchIndex(),
+ expectedSeekGeneration);
+ lastWalGapWaitLogTimeMs = nowMs;
+ }
+ onWalGapRetryScheduled();
+ pendingWalGapRetryRequested = true;
+ return false;
+ }
+
+ /**
+ * Try catch-up from WAL when the pending queue was empty. This handles cold-start or scenarios
+ * where the subscription started after data was already written.
+ */
+ private void tryCatchUpFromWAL(final long expectedSeekGeneration) {
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+
+ final DeliveryBatchState batchState = new DeliveryBatchState();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ final boolean accepted =
+ pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, maxWalEntries, maxTablets, maxBatchBytes);
+ if (!accepted) {
+ return;
+ }
+
+ if (!batchState.isEmpty()) {
+ flushBatch(batchState, expectedSeekGeneration);
+ }
+ }
+
+ private boolean pumpFromSubscriptionWAL(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxWalEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return true;
+ }
+
+ subscriptionWALIterator.refresh();
+ ensureSubscriptionWalReadable();
+
+ int entriesRead = 0;
+ while (entriesRead < maxWalEntries
+ && subscriptionWALIterator.hasNext()
+ && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ try {
+ final IndexedConsensusRequest walEntry = subscriptionWALIterator.next();
+ entriesRead++;
+
+ if (isBeforeLocalCursor(walEntry)) {
+ continue;
+ }
+ if (shouldSkipForRecoveryProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ walEntry, batchState, expectedSeekGeneration, maxTablets, maxBatchBytes, false)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(walEntry);
+ advanceLocalCursorIfPresent(walEntry);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error reading subscription WAL", this, e);
+ break;
+ }
+ }
+
+ if (entriesRead > 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL read {} entries, nextExpectedSearchIndex={}",
+ this,
+ entriesRead,
+ nextExpectedSearchIndex.get());
+ }
+ return true;
+ }
+
+ private void ensureSubscriptionWalReadable() {
+ if (Objects.isNull(subscriptionWALIterator)
+ || subscriptionWALIterator.hasNext()
+ || !(consensusReqReader instanceof WALNode)) {
+ return;
+ }
+
+ final long currentWalIndex = consensusReqReader.getCurrentSearchIndex();
+ if (nextExpectedSearchIndex.get() > currentWalIndex) {
+ return;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL exhausted at {} while current WAL is {}. "
+ + "Rolling WAL file to expose current-file entries.",
+ this,
+ nextExpectedSearchIndex.get(),
+ currentWalIndex);
+ ((WALNode) consensusReqReader).rollWALFile();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ if (Objects.nonNull(subscriptionWALIterator)) {
+ subscriptionWALIterator.refresh();
+ }
+ }
+
+ private void resetSubscriptionWALPosition(final long startSearchIndex) {
+ closeSubscriptionWALIterator();
+ subscriptionWALIterator = createSubscriptionWALIterator(startSearchIndex);
+ }
+
+ protected ProgressWALIterator createSubscriptionWALIterator(final long startSearchIndex) {
+ if (consensusReqReader instanceof WALNode) {
+ return new ProgressWALIterator((WALNode) consensusReqReader, startSearchIndex);
+ }
+ return null;
+ }
+
+ protected void onWalGapRetryScheduled() {}
+
+ private boolean hasReadableWalEntries() {
+ return Objects.nonNull(subscriptionWALIterator) && subscriptionWALIterator.hasNext();
+ }
+
+ private void requestSubscriptionWalReset(
+ final long targetSearchIndex, final long seekGenerationValue) {
+ pendingSubscriptionWalResetSearchIndex = targetSearchIndex;
+ pendingSubscriptionWalResetGeneration = seekGenerationValue;
+ }
+
+ private void applyPendingSubscriptionWalReset(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ resetSubscriptionWALPosition(pendingSubscriptionWalResetSearchIndex);
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ }
+
+ private void restorePendingSubscriptionWalCursor(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ // A seek can land in the middle of a prefetch iteration. Restore the local cursor to the
+ // pending seek target before resuming under the new generation so stale in-flight work does
+ // not permanently advance the historical replay frontier.
+ nextExpectedSearchIndex.set(pendingSubscriptionWalResetSearchIndex);
+ }
+
+ private void closeSubscriptionWALIterator() {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return;
+ }
+ try {
+ subscriptionWALIterator.close();
+ } catch (final IOException e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error closing subscription WAL iterator", this, e);
+ } finally {
+ subscriptionWALIterator = null;
+ }
+ }
+
+ /**
+ * Deserializes the IConsensusRequest entries within an IndexedConsensusRequest to produce an
+ * InsertNode. WAL entries are typically stored as IoTConsensusRequest (serialized ByteBuffers),
+ * and a single logical write may be split across multiple fragments (SearchNode). This method
+ * handles both cases.
+ *
+ * <p>The deserialization follows the same pattern as {@code
+ * DataRegionStateMachine.grabPlanNode()}.
+ */
+ private InsertNode deserializeToInsertNode(final IndexedConsensusRequest indexedRequest) {
+ final List<SearchNode> searchNodes = new ArrayList<>();
+ PlanNode nonSearchNode = null;
+
+ for (final IConsensusRequest req : indexedRequest.getRequests()) {
+ PlanNode planNode;
+ try {
+ if (req instanceof IoTConsensusRequest) {
+ // WAL entries read from file are wrapped as IoTConsensusRequest (ByteBuffer)
+ planNode = WALEntry.deserializeForConsensus(req.serializeToByteBuffer());
+ } else if (req instanceof InsertNode) {
+ // In-memory entries (not yet flushed to WAL file) may already be PlanNode
+ planNode = (PlanNode) req;
+ } else {
+ // ByteBufferConsensusRequest or unknown
+ planNode = PlanNodeType.deserialize(req.serializeToByteBuffer());
+ }
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to deserialize IConsensusRequest "
+ + "(type={}) in searchIndex={}: {}",
+ this,
+ req.getClass().getSimpleName(),
+ indexedRequest.getSearchIndex(),
+ e.getMessage(),
+ e);
+ continue;
+ }
+
+ if (planNode instanceof SearchNode) {
+ final SearchNode searchNode = (SearchNode) planNode;
+ searchNode.setSearchIndex(indexedRequest.getSearchIndex());
+ if (indexedRequest.getSyncIndex() >= 0) {
+ searchNode.setSyncIndex(indexedRequest.getSyncIndex());
+ }
+ if (indexedRequest.getPhysicalTime() > 0) {
+ searchNode.setPhysicalTime(indexedRequest.getPhysicalTime());
+ }
+ if (indexedRequest.getNodeId() >= 0) {
+ searchNode.setNodeId(indexedRequest.getNodeId());
+ }
+ searchNodes.add(searchNode);
+ } else {
+ nonSearchNode = planNode;
+ }
+ }
+
+ // Merge split SearchNode fragments (same pattern as DataRegionStateMachine.grabPlanNode)
+ if (!searchNodes.isEmpty()) {
+ final PlanNode merged = searchNodes.get(0).merge(searchNodes);
+ if (merged instanceof InsertNode) {
+ final InsertNode mergedInsert = (InsertNode) merged;
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: deserialized merged InsertNode for searchIndex={}, "
+ + "type={}, deviceId={}, searchNodeCount={}",
+ this,
+ indexedRequest.getSearchIndex(),
+ mergedInsert.getType(),
+ ConsensusLogToTabletConverter.safeDeviceIdForLog(mergedInsert),
+ searchNodes.size());
+
+ return mergedInsert;
+ }
+ }
+
+ if (nonSearchNode != null) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: searchIndex={} contains non-InsertNode PlanNode: {}",
+ this,
+ indexedRequest.getSearchIndex(),
+ nonSearchNode.getClass().getSimpleName());
+ }
+
+ return null;
+ }
+
+ private PreparedEntry prepareEntry(final IndexedConsensusRequest indexedRequest) {
+ final InsertNode insertNode = deserializeToInsertNode(indexedRequest);
+ if (Objects.isNull(insertNode)) {
+ return null;
+ }
+
+ final long localSeq =
+ indexedRequest.getProgressLocalSeq() >= 0
+ ? indexedRequest.getProgressLocalSeq()
+ : indexedRequest.getSearchIndex();
+ final long searchIndex = indexedRequest.getSearchIndex();
+ final long physicalTime =
+ indexedRequest.getPhysicalTime() > 0
+ ? indexedRequest.getPhysicalTime()
+ : insertNode.getPhysicalTime();
+ final int writerNodeId =
+ indexedRequest.getNodeId() >= 0 ? indexedRequest.getNodeId() : insertNode.getNodeId();
+
+ trackWriterLane(writerNodeId);
+ final long maxTs = extractMaxTime(insertNode);
+ if (maxTs > maxObservedTimestamp) {
+ maxObservedTimestamp = maxTs;
+ }
+ final List<Tablet> tablets = converter.convert(insertNode);
+ if (tablets.isEmpty()) {
+ return null;
+ }
+
+ return new PreparedEntry(tablets, physicalTime, writerNodeId, localSeq, searchIndex);
+ }
+
+ private static long estimateTabletSize(final Tablet tablet) {
+ return PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet);
+ }
+
+ private void createAndEnqueueEvent(
+ final List<Tablet> tablets, final long startSearchIndex, final long endSearchIndex) {
+ createAndEnqueueEvent(
+ tablets, startSearchIndex, endSearchIndex, endSearchIndex, seekGeneration.get());
+ }
+
+ private boolean createAndEnqueueEvent(
+ final List<Tablet> tablets,
+ final long startSearchIndex,
+ final long endSearchIndex,
+ final long commitLocalSeq,
+ final long expectedSeekGeneration) {
+ if (tablets.isEmpty()) {
+ return true;
+ }
+
+ if (seekGeneration.get() != expectedSeekGeneration) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: skip stale event with searchIndex range [{}, {}], "
+ + "expectedSeekGeneration={}, currentSeekGeneration={}",
+ this,
+ startSearchIndex,
+ endSearchIndex,
+ expectedSeekGeneration,
+ seekGeneration.get());
+ return false;
+ }
+
+ final SubscriptionCommitContext commitContext = buildWriterCommitContext(commitLocalSeq);
+ final WriterId writerId = commitContext.getWriterId();
+ final WriterProgress writerProgress = commitContext.getWriterProgress();
+ commitManager.recordMapping(brokerId, topicName, consensusGroupId, writerId, writerProgress);
+
+ // nextOffset <= 0 means all tablets delivered in single batch
+ // -tablets.size() indicates total count
+ // Use Map<String, List<Tablet>> constructor with actual database name for table model;
+ final TabletsPayload payload =
+ new TabletsPayload(
+ Collections.singletonMap(converter.getDatabaseName(), tablets), -tablets.size());
+
+ final SubscriptionEvent event =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.TABLETS.getType(), payload, commitContext);
+
+ prefetchingQueue.add(event);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: ENQUEUED event with {} tablets, "
+ + "searchIndex range [{}, {}], prefetchQueueSize={}",
+ this,
+ tablets.size(),
+ startSearchIndex,
+ endSearchIndex,
+ prefetchingQueue.size());
+
+ // After enqueuing the data event, control metadata is handled separately from user data.
+ return true;
+ }
+
+ private SubscriptionCommitContext buildWriterCommitContext(final long localSeq) {
+ final int effectiveNodeId =
+ batchWriterNodeId >= 0
+ ? batchWriterNodeId
+ : IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), effectiveNodeId);
+ final WriterProgress writerProgress = new WriterProgress(batchPhysicalTime, localSeq);
+ return new SubscriptionCommitContext(
+ IoTDBDescriptor.getInstance().getConfig().getDataNodeId(),
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ seekGeneration.get(),
+ writerId,
+ writerProgress);
+ }
+
+ private void updateBatchWriterProgress(final long physicalTime, final int writerNodeId) {
+ if (physicalTime > 0) {
+ this.batchPhysicalTime = physicalTime;
+ }
+ if (writerNodeId >= 0) {
+ this.batchWriterNodeId = writerNodeId;
+ }
+ }
+
+ private void resetBatchWriterProgress() {
+ this.batchPhysicalTime = 0L;
+ this.batchWriterNodeId = -1;
+ }
+
+ private long estimateTabletsBytes(final List<Tablet> tablets) {
+ long estimatedBytes = 0L;
+ for (final Tablet tablet : tablets) {
+ estimatedBytes += estimateTabletSize(tablet);
+ }
+ return estimatedBytes;
+ }
+
+ private boolean appendPreparedEntryViaRealtimeLane(
+ final DeliveryBatchState batchState,
+ final PreparedEntry preparedEntry,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ bufferRealtimeEntry(preparedEntry);
+ return drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes);
+ }
+
+ private int getRealtimeBufferedEntryCount() {
+ int count = 0;
+ for (final NavigableMap<Long, PreparedEntry> laneEntries : realtimeEntriesByLane.values()) {
+ count += laneEntries.size();
+ }
+ return count;
+ }
+
+ private boolean drainBufferedRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ while (!realtimeEntriesByLane.isEmpty()) {
+ final int bufferedBefore = getRealtimeBufferedEntryCount();
+ if (!drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final int bufferedAfter = getRealtimeBufferedEntryCount();
+ if (bufferedAfter == 0 || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return true;
+ }
+
+ if (batchState.isEmpty()) {
+ return true;
+ }
+
+ if (!flushBatch(batchState, expectedSeekGeneration)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private boolean canAppendLaneEntry(
+ final DeliveryBatchState batchState,
+ final LaneBufferedEntry entry,
+ final long entryEstimatedBytes,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ final boolean wouldExceedEntryLimit =
+ maxEntries != Integer.MAX_VALUE && batchState.entryCount >= maxEntries;
+ final boolean wouldExceedTabletLimit =
+ !batchState.isEmpty() && batchState.tablets.size() + entry.getTablets().size() > maxTablets;
+ final boolean wouldExceedByteLimit =
+ !batchState.isEmpty() && batchState.estimatedBytes + entryEstimatedBytes > maxBatchBytes;
+ // Keep all consensus subscription modes on a single-writer commit/delivery shape so
+ // SubscriptionCommitContext and RegionProgress remain per-writer.
+ final boolean writerChanged =
+ !batchState.isEmpty() && batchState.writerNodeId != entry.getWriterNodeId();
+ return !(wouldExceedEntryLimit
+ || wouldExceedTabletLimit
+ || wouldExceedByteLimit
+ || writerChanged);
+ }
+
+ private boolean drainRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ return drainLaneEntries(
+ batchState,
+ this::buildRealtimeLaneFrontiers,
+ this::peekRealtimeEntry,
+ entry -> true,
+ (laneId, entry) -> removeRealtimeEntry(laneId, entry.localSeq),
+ Integer.MAX_VALUE,
+ maxTablets,
+ maxBatchBytes,
+ true);
+ }
+
+ private <T extends LaneBufferedEntry> boolean drainLaneEntries(
+ final DeliveryBatchState batchState,
+ final Supplier<PriorityQueue<LaneFrontier>> frontierSupplier,
+ final Function<WriterLaneId, T> headSupplier,
+ final Predicate<T> releasePredicate,
+ final BiConsumer<WriterLaneId, T> removeHeadAction,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean trackLingerTime) {
+ while (true) {
+ final PriorityQueue<LaneFrontier> frontiers = frontierSupplier.get();
+ if (frontiers.isEmpty()) {
+ return true;
+ }
+ final LaneFrontier frontier = frontiers.peek();
+ if (Objects.isNull(frontier) || frontier.isBarrier) {
+ return true;
+ }
+ final T laneHead = headSupplier.apply(frontier.laneId);
+ if (Objects.isNull(laneHead)) {
+ return true;
+ }
+ if (!releasePredicate.test(laneHead)) {
+ return true;
+ }
+
+ final long entryEstimatedBytes = estimateTabletsBytes(laneHead.getTablets());
+ if (!canAppendLaneEntry(
+ batchState, laneHead, entryEstimatedBytes, maxEntries, maxTablets, maxBatchBytes)) {
+ return true;
+ }
+
+ removeHeadAction.accept(frontier.laneId, laneHead);
+ batchState.append(laneHead, entryEstimatedBytes, trackLingerTime);
+ }
+ }
+
+ private boolean flushBatch(
+ final DeliveryBatchState batchState, final long expectedSeekGeneration) {
+ updateBatchWriterProgress(batchState.physicalTime, batchState.writerNodeId);
+ if (!createAndEnqueueEvent(
+ new ArrayList<>(batchState.tablets),
+ batchState.startSearchIndex,
+ batchState.endSearchIndex,
+ batchState.lastLocalSeq,
+ expectedSeekGeneration)) {
+ return false;
+ }
+ resetBatchWriterProgress();
+ batchState.reset();
+ return true;
+ }
+
+ // ======================== Commit (Ack/Nack) ========================
+
+ private boolean canAcceptCommitContext(
+ final SubscriptionCommitContext commitContext, final String action, final boolean silent) {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+ if (Objects.isNull(commitContext) || !commitContext.hasWriterProgress()) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ }
+ return false;
+ }
+ if (!isActive) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ }
+ return false;
+ }
+ return true;
+ }
+
+ public boolean ack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "ack", false)
+ && ackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean ackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ if (!acked.get()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for ack",
+ this,
+ commitContext);
+ }
+ return null;
+ }
+
+ if (ev.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: event {} already committed", this, commitContext);
+ ev.cleanUp(false);
+ return null;
+ }
+
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to advance commit frontier for {}",
+ this,
+ commitContext);
+ return ev;
+ }
+
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+
+ return acked.get();
+ }
+
+ public boolean nack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "nack", false)
+ && nackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ /**
+ * Silent version of ack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean ackSilent(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "ack", true)) {
+ return false;
+ }
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ return ev;
+ }
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+ return acked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private WriterId extractCommitWriterId(final SubscriptionCommitContext commitContext) {
+ final WriterId writerId = commitContext.getWriterId();
+ return Objects.nonNull(writerId) ? writerId : new WriterId(consensusGroupId.toString(), -1);
+ }
+
+ private WriterProgress extractCommitWriterProgress(
+ final SubscriptionCommitContext commitContext) {
+ return commitContext.getWriterProgress();
+ }
+
+ /**
+ * Silent version of nack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean nackSilent(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "nack", true)) {
+ return false;
+ }
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+ return nacked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean nackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for nack",
+ this,
+ commitContext);
+ return null;
+ }
+
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+
+ return nacked.get();
+ }
+
+ // ======================== Recycle ========================
+
+ /** Recycles in-flight events that are pollable (timed out) back to the prefetching queue. */
+ private void recycleInFlightEvents() {
+ for (final Pair<String, SubscriptionCommitContext> key :
+ new ArrayList<>(inFlightEvents.keySet())) {
+ inFlightEvents.compute(
+ key,
+ (k, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ if (ev.pollable()) {
+ ev.nack();
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected during recycle "
+ + "(nackCount={}), force-acking event {}",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: recycled timed-out event {} back to prefetching queue",
+ this,
+ ev);
+ return null;
+ }
+ return ev;
+ });
+ }
+ }
+
+ // ======================== Cleanup ========================
+
+ public void cleanUp() {
+ acquireWriteLock();
+ try {
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ pendingEntries.clear();
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ closeSubscriptionWALIterator();
+
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ // ======================== Seek ========================
+
+ /**
+ * Seeks to the earliest available WAL position. The actual position depends on WAL retention: if
+ * old files have been reclaimed, the earliest available position may be later than 0.
+ */
+ public void seekToBeginning() {
+ seekToResolvedPosition(0L, new RegionProgress(Collections.emptyMap()), "beginning");
+ }
+
+ /**
+ * Seeks to the current WAL write position. After this, only newly written data will be consumed.
+ */
+ public void seekToEnd() {
+ seekToResolvedPosition(
+ consensusReqReader.getCurrentSearchIndex(), computeTailRegionProgress(), "end");
+ }
+
+ public void seekToRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress not supported (no WAL directory)",
+ this);
+ seekToBeginning();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, false);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgress");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekToRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ public void seekAfterRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress not supported (no WAL directory)",
+ this);
+ seekToEnd();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgressAfter");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekAfterRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
Review Comment:
Added a comment clarifying that replay lookup normalizes the target to the beginning, and the seek reset moves the WAL cursor there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279209875
##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties:
##########
@@ -54,6 +54,12 @@ dn_data_region_consensus_port=10760
schema_replication_factor=1
data_replication_factor=1
+####################
+### Subscription Consensus Configuration
+####################
+
+# subscription_consensus_idle_safe_hlc_interval_ms=10000
+
Review Comment:
Moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2922758960
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionConsumer.java:
##########
@@ -1373,6 +1412,44 @@ private void unsubscribeWithRedirection(final Set<String> topicNames)
throw new SubscriptionRuntimeCriticalException(errorMessage);
}
+ /**
+ * Sends seek request to ALL available providers. Unlike subscribe/unsubscribe, seek must reach
+ * every node because data regions for the topic may be distributed across different nodes.
+ */
+ private void seekWithRedirection(
+ final String topicName, final short seekType, final long timestamp)
+ throws SubscriptionException {
+ final List<AbstractSubscriptionProvider> providers = this.providers.getAllAvailableProviders();
+ if (providers.isEmpty()) {
+ throw new SubscriptionConnectionException(
+ String.format(
+ "Cluster has no available subscription providers when %s seek topic %s",
+ this, topicName));
+ }
+ boolean anySuccess = false;
+ for (final AbstractSubscriptionProvider provider : providers) {
+ try {
+ provider.seek(topicName, seekType, timestamp);
+ anySuccess = true;
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "{} failed to seek topic {} from subscription provider {}, continuing with other providers...",
+ this,
+ topicName,
+ provider,
+ e);
+ }
+ }
+ if (!anySuccess) {
Review Comment:
Shall we throw exception for "anySuccess" or "allSuccess" since "seek must reach every node"?
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/subscription/subscription/CreateSubscriptionProcedure.java:
##########
@@ -103,15 +107,41 @@ protected boolean executeFromValidate(final ConfigNodeProcedureEnv env)
alterConsumerGroupProcedure =
new AlterConsumerGroupProcedure(updatedConsumerGroupMeta, subscriptionInfo);
- // Construct CreatePipeProcedureV2s
+ // Construct CreatePipeProcedureV2s (for non-consensus topics)
for (final String topicName : subscribeReq.getTopicNames()) {
+ final TopicMeta topicMeta = subscriptionInfo.get().deepCopyTopicMeta(topicName);
+
+ // Check if this topic should use consensus subscription: mode is live, format is Tablet
+ final String topicMode =
+ topicMeta
+ .getConfig()
+ .getStringOrDefault(TopicConstant.MODE_KEY, TopicConstant.MODE_DEFAULT_VALUE);
+ final String topicFormat =
+ topicMeta
+ .getConfig()
+ .getStringOrDefault(TopicConstant.FORMAT_KEY, TopicConstant.FORMAT_DEFAULT_VALUE);
+ final boolean isConsensusBasedTopic =
+ TopicConstant.MODE_LIVE_VALUE.equalsIgnoreCase(topicMode)
+ && !TopicConstant.FORMAT_TS_FILE_HANDLER_VALUE.equalsIgnoreCase(topicFormat);
Review Comment:
Shall MODE_LIVE + TS_FILE_HANDLER throw an exception here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2923077979
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/ConsensusSubscriptionBroker.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker;
+
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusLogToTabletConverter;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusPrefetchingQueue;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusSubscriptionCommitManager;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.request.PipeSubscribeSeekReq;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * Consensus-based subscription broker that reads data directly from IoTConsensus WAL. Each instance
+ * manages consensus prefetching queues for a single consumer group.
+ */
+public class ConsensusSubscriptionBroker implements ISubscriptionBroker {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusSubscriptionBroker.class);
+
+ private final String brokerId; // consumer group id
+
+ /** Maps topic name to a list of ConsensusPrefetchingQueues, one per data region. */
+ private final Map<String, List<ConsensusPrefetchingQueue>> topicNameToConsensusPrefetchingQueues;
+
+ /** Shared commit ID generators per topic. */
+ private final Map<String, AtomicLong> topicNameToCommitIdGenerator;
+
+ public ConsensusSubscriptionBroker(final String brokerId) {
+ this.brokerId = brokerId;
+ this.topicNameToConsensusPrefetchingQueues = new ConcurrentHashMap<>();
+ this.topicNameToCommitIdGenerator = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return topicNameToConsensusPrefetchingQueues.isEmpty();
+ }
+
+ @Override
+ public boolean hasQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ return Objects.nonNull(queues)
+ && !queues.isEmpty()
+ && queues.stream().anyMatch(q -> !q.isClosed());
+ }
+
+ //////////////////////////// poll ////////////////////////////
+
+ @Override
+ public List<SubscriptionEvent> poll(
+ final String consumerId, final Set<String> topicNames, final long maxBytes) {
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll called, consumerId={}, topicNames={}, "
+ + "queueCount={}, maxBytes={}",
+ brokerId,
+ consumerId,
+ topicNames,
+ topicNameToConsensusPrefetchingQueues.size(),
+ maxBytes);
+
+ final List<SubscriptionEvent> eventsToPoll = new ArrayList<>();
+ final List<SubscriptionEvent> eventsToNack = new ArrayList<>();
+ long totalSize = 0;
+
+ for (final String topicName : topicNames) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ continue;
+ }
+
+ // Poll from all region queues for this topic
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+
+ final SubscriptionEvent event = consensusQueue.poll(consumerId);
+ if (Objects.isNull(event)) {
+ continue;
+ }
+
+ final long currentSize;
+ try {
+ currentSize = event.getCurrentResponseSize();
+ } catch (final IOException e) {
+ eventsToNack.add(event);
+ continue;
+ }
+
+ eventsToPoll.add(event);
+ totalSize += currentSize;
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ // Nack any events that had errors
+ if (!eventsToNack.isEmpty()) {
+ commit(
+ consumerId,
+ eventsToNack.stream()
+ .map(SubscriptionEvent::getCommitContext)
+ .collect(Collectors.toList()),
+ true);
+ }
+
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll result, consumerId={}, eventsPolled={}, eventsNacked={}",
+ brokerId,
+ consumerId,
+ eventsToPoll.size(),
+ eventsToNack.size());
+
+ return eventsToPoll;
+ }
+
+ @Override
+ public List<SubscriptionEvent> pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Try each region queue until one returns a match
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+ final SubscriptionEvent event = consensusQueue.pollTablets(consumerId, commitContext, offset);
+ if (Objects.nonNull(event)) {
+ return Collections.singletonList(event);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ //////////////////////////// commit ////////////////////////////
+
+ @Override
+ public List<SubscriptionCommitContext> commit(
+ final String consumerId,
+ final List<SubscriptionCommitContext> commitContexts,
+ final boolean nack) {
+ final List<SubscriptionCommitContext> successfulCommitContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext commitContext : commitContexts) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to commit",
+ brokerId,
+ topicName);
+ continue;
+ }
+
+ // Try each region queue for this topic (the event belongs to exactly one region).
+ // Don't warn per-queue miss — only warn if NO queue handled the commit.
+ boolean handled = false;
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+ final boolean success;
+ if (!nack) {
+ success = consensusQueue.ackSilent(consumerId, commitContext);
+ } else {
+ success = consensusQueue.nackSilent(consumerId, commitContext);
+ }
+ if (success) {
+ successfulCommitContexts.add(commitContext);
+ handled = true;
+ break; // committed in the right queue, no need to try others
+ }
+ }
+ if (!handled) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: commit context {} not found in any of {} region queue(s) for topic [{}]",
+ brokerId,
+ commitContext,
+ queues.size(),
+ topicName);
+ }
+ }
+ return successfulCommitContexts;
+ }
+
+ @Override
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return true;
+ }
+ // Any queue that considers it NOT outdated means it's not outdated
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!q.isCommitContextOutdated(commitContext)) {
Review Comment:
Shall a context be compared with all the queues, while it only belongs to one?
However, it's OK not to judge it, since now it only affects logger and idempotent "NACK"...
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/ConsensusSubscriptionBroker.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker;
+
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusLogToTabletConverter;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusPrefetchingQueue;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusSubscriptionCommitManager;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.request.PipeSubscribeSeekReq;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * Consensus-based subscription broker that reads data directly from IoTConsensus WAL. Each instance
+ * manages consensus prefetching queues for a single consumer group.
+ */
+public class ConsensusSubscriptionBroker implements ISubscriptionBroker {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusSubscriptionBroker.class);
+
+ private final String brokerId; // consumer group id
+
+ /** Maps topic name to a list of ConsensusPrefetchingQueues, one per data region. */
+ private final Map<String, List<ConsensusPrefetchingQueue>> topicNameToConsensusPrefetchingQueues;
+
+ /** Shared commit ID generators per topic. */
+ private final Map<String, AtomicLong> topicNameToCommitIdGenerator;
+
+ public ConsensusSubscriptionBroker(final String brokerId) {
+ this.brokerId = brokerId;
+ this.topicNameToConsensusPrefetchingQueues = new ConcurrentHashMap<>();
+ this.topicNameToCommitIdGenerator = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return topicNameToConsensusPrefetchingQueues.isEmpty();
+ }
+
+ @Override
+ public boolean hasQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ return Objects.nonNull(queues)
+ && !queues.isEmpty()
+ && queues.stream().anyMatch(q -> !q.isClosed());
+ }
+
+ //////////////////////////// poll ////////////////////////////
+
+ @Override
+ public List<SubscriptionEvent> poll(
+ final String consumerId, final Set<String> topicNames, final long maxBytes) {
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll called, consumerId={}, topicNames={}, "
+ + "queueCount={}, maxBytes={}",
+ brokerId,
+ consumerId,
+ topicNames,
+ topicNameToConsensusPrefetchingQueues.size(),
+ maxBytes);
+
+ final List<SubscriptionEvent> eventsToPoll = new ArrayList<>();
+ final List<SubscriptionEvent> eventsToNack = new ArrayList<>();
+ long totalSize = 0;
+
+ for (final String topicName : topicNames) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ continue;
+ }
+
+ // Poll from all region queues for this topic
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+
+ final SubscriptionEvent event = consensusQueue.poll(consumerId);
+ if (Objects.isNull(event)) {
+ continue;
+ }
+
+ final long currentSize;
+ try {
+ currentSize = event.getCurrentResponseSize();
+ } catch (final IOException e) {
+ eventsToNack.add(event);
+ continue;
+ }
+
+ eventsToPoll.add(event);
+ totalSize += currentSize;
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ // Nack any events that had errors
+ if (!eventsToNack.isEmpty()) {
+ commit(
+ consumerId,
+ eventsToNack.stream()
+ .map(SubscriptionEvent::getCommitContext)
+ .collect(Collectors.toList()),
+ true);
+ }
+
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll result, consumerId={}, eventsPolled={}, eventsNacked={}",
+ brokerId,
+ consumerId,
+ eventsToPoll.size(),
+ eventsToNack.size());
+
+ return eventsToPoll;
+ }
+
+ @Override
+ public List<SubscriptionEvent> pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Try each region queue until one returns a match
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+ final SubscriptionEvent event = consensusQueue.pollTablets(consumerId, commitContext, offset);
+ if (Objects.nonNull(event)) {
+ return Collections.singletonList(event);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ //////////////////////////// commit ////////////////////////////
+
+ @Override
+ public List<SubscriptionCommitContext> commit(
+ final String consumerId,
+ final List<SubscriptionCommitContext> commitContexts,
+ final boolean nack) {
+ final List<SubscriptionCommitContext> successfulCommitContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext commitContext : commitContexts) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to commit",
+ brokerId,
+ topicName);
+ continue;
+ }
+
+ // Try each region queue for this topic (the event belongs to exactly one region).
+ // Don't warn per-queue miss — only warn if NO queue handled the commit.
+ boolean handled = false;
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+ final boolean success;
+ if (!nack) {
+ success = consensusQueue.ackSilent(consumerId, commitContext);
+ } else {
+ success = consensusQueue.nackSilent(consumerId, commitContext);
+ }
+ if (success) {
+ successfulCommitContexts.add(commitContext);
+ handled = true;
+ break; // committed in the right queue, no need to try others
+ }
+ }
+ if (!handled) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: commit context {} not found in any of {} region queue(s) for topic [{}]",
+ brokerId,
+ commitContext,
+ queues.size(),
+ topicName);
+ }
+ }
+ return successfulCommitContexts;
+ }
+
+ @Override
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return true;
+ }
+ // Any queue that considers it NOT outdated means it's not outdated
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!q.isCommitContextOutdated(commitContext)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ //////////////////////////// seek ////////////////////////////
+
+ public void seek(final String topicName, final short seekType, final long timestamp) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to seek",
+ brokerId,
+ topicName);
+ return;
+ }
+
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ switch (seekType) {
+ case PipeSubscribeSeekReq.SEEK_TO_BEGINNING:
+ queue.seekToBeginning();
+ break;
+ case PipeSubscribeSeekReq.SEEK_TO_END:
+ queue.seekToEnd();
+ break;
+ case PipeSubscribeSeekReq.SEEK_TO_TIMESTAMP:
+ queue.seekToTimestamp(timestamp);
+ break;
+ default:
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: unknown seekType {} for topic [{}]",
+ brokerId,
+ seekType,
+ topicName);
+ break;
+ }
+ }
+ }
+
+ //////////////////////////// prefetching ////////////////////////////
+
+ @Override
+ public boolean executePrefetch(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return false;
+ }
+ boolean anyPrefetched = false;
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!q.isClosed() && q.executePrefetch()) {
+ anyPrefetched = true;
+ }
+ }
+ return anyPrefetched;
+ }
+
+ @Override
+ public int getEventCount(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues)) {
+ return 0;
+ }
+ return queues.stream().mapToInt(ConsensusPrefetchingQueue::getPrefetchedEventCount).sum();
Review Comment:
May filter the closed ones here?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/ConsensusSubscriptionBroker.java:
##########
@@ -0,0 +1,432 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker;
+
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusLogToTabletConverter;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusPrefetchingQueue;
+import org.apache.iotdb.db.subscription.broker.consensus.ConsensusSubscriptionCommitManager;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.request.PipeSubscribeSeekReq;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.stream.Collectors;
+
+/**
+ * Consensus-based subscription broker that reads data directly from IoTConsensus WAL. Each instance
+ * manages consensus prefetching queues for a single consumer group.
+ */
+public class ConsensusSubscriptionBroker implements ISubscriptionBroker {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusSubscriptionBroker.class);
+
+ private final String brokerId; // consumer group id
+
+ /** Maps topic name to a list of ConsensusPrefetchingQueues, one per data region. */
+ private final Map<String, List<ConsensusPrefetchingQueue>> topicNameToConsensusPrefetchingQueues;
+
+ /** Shared commit ID generators per topic. */
+ private final Map<String, AtomicLong> topicNameToCommitIdGenerator;
+
+ public ConsensusSubscriptionBroker(final String brokerId) {
+ this.brokerId = brokerId;
+ this.topicNameToConsensusPrefetchingQueues = new ConcurrentHashMap<>();
+ this.topicNameToCommitIdGenerator = new ConcurrentHashMap<>();
+ }
+
+ @Override
+ public boolean isEmpty() {
+ return topicNameToConsensusPrefetchingQueues.isEmpty();
+ }
+
+ @Override
+ public boolean hasQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ return Objects.nonNull(queues)
+ && !queues.isEmpty()
+ && queues.stream().anyMatch(q -> !q.isClosed());
+ }
+
+ //////////////////////////// poll ////////////////////////////
+
+ @Override
+ public List<SubscriptionEvent> poll(
+ final String consumerId, final Set<String> topicNames, final long maxBytes) {
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll called, consumerId={}, topicNames={}, "
+ + "queueCount={}, maxBytes={}",
+ brokerId,
+ consumerId,
+ topicNames,
+ topicNameToConsensusPrefetchingQueues.size(),
+ maxBytes);
+
+ final List<SubscriptionEvent> eventsToPoll = new ArrayList<>();
+ final List<SubscriptionEvent> eventsToNack = new ArrayList<>();
+ long totalSize = 0;
+
+ for (final String topicName : topicNames) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ continue;
+ }
+
+ // Poll from all region queues for this topic
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+
+ final SubscriptionEvent event = consensusQueue.poll(consumerId);
+ if (Objects.isNull(event)) {
+ continue;
+ }
+
+ final long currentSize;
+ try {
+ currentSize = event.getCurrentResponseSize();
+ } catch (final IOException e) {
+ eventsToNack.add(event);
+ continue;
+ }
+
+ eventsToPoll.add(event);
+ totalSize += currentSize;
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ if (totalSize >= maxBytes) {
+ break;
+ }
+ }
+
+ // Nack any events that had errors
+ if (!eventsToNack.isEmpty()) {
+ commit(
+ consumerId,
+ eventsToNack.stream()
+ .map(SubscriptionEvent::getCommitContext)
+ .collect(Collectors.toList()),
+ true);
+ }
+
+ LOGGER.debug(
+ "ConsensusSubscriptionBroker [{}]: poll result, consumerId={}, eventsPolled={}, eventsNacked={}",
+ brokerId,
+ consumerId,
+ eventsToPoll.size(),
+ eventsToNack.size());
+
+ return eventsToPoll;
+ }
+
+ @Override
+ public List<SubscriptionEvent> pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Try each region queue until one returns a match
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+ final SubscriptionEvent event = consensusQueue.pollTablets(consumerId, commitContext, offset);
+ if (Objects.nonNull(event)) {
+ return Collections.singletonList(event);
+ }
+ }
+ return Collections.emptyList();
+ }
+
+ //////////////////////////// commit ////////////////////////////
+
+ @Override
+ public List<SubscriptionCommitContext> commit(
+ final String consumerId,
+ final List<SubscriptionCommitContext> commitContexts,
+ final boolean nack) {
+ final List<SubscriptionCommitContext> successfulCommitContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext commitContext : commitContexts) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to commit",
+ brokerId,
+ topicName);
+ continue;
+ }
+
+ // Try each region queue for this topic (the event belongs to exactly one region).
+ // Don't warn per-queue miss — only warn if NO queue handled the commit.
+ boolean handled = false;
+ for (final ConsensusPrefetchingQueue consensusQueue : queues) {
+ if (consensusQueue.isClosed()) {
+ continue;
+ }
+ final boolean success;
+ if (!nack) {
+ success = consensusQueue.ackSilent(consumerId, commitContext);
+ } else {
+ success = consensusQueue.nackSilent(consumerId, commitContext);
+ }
+ if (success) {
+ successfulCommitContexts.add(commitContext);
+ handled = true;
+ break; // committed in the right queue, no need to try others
+ }
+ }
+ if (!handled) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: commit context {} not found in any of {} region queue(s) for topic [{}]",
+ brokerId,
+ commitContext,
+ queues.size(),
+ topicName);
+ }
+ }
+ return successfulCommitContexts;
+ }
+
+ @Override
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ final String topicName = commitContext.getTopicName();
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return true;
+ }
+ // Any queue that considers it NOT outdated means it's not outdated
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!q.isCommitContextOutdated(commitContext)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ //////////////////////////// seek ////////////////////////////
+
+ public void seek(final String topicName, final short seekType, final long timestamp) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: no queues for topic [{}] to seek",
+ brokerId,
+ topicName);
+ return;
+ }
+
+ for (final ConsensusPrefetchingQueue queue : queues) {
+ if (queue.isClosed()) {
+ continue;
+ }
+ switch (seekType) {
+ case PipeSubscribeSeekReq.SEEK_TO_BEGINNING:
+ queue.seekToBeginning();
+ break;
+ case PipeSubscribeSeekReq.SEEK_TO_END:
+ queue.seekToEnd();
+ break;
+ case PipeSubscribeSeekReq.SEEK_TO_TIMESTAMP:
+ queue.seekToTimestamp(timestamp);
+ break;
+ default:
+ LOGGER.warn(
+ "ConsensusSubscriptionBroker [{}]: unknown seekType {} for topic [{}]",
+ brokerId,
+ seekType,
+ topicName);
+ break;
+ }
+ }
+ }
+
+ //////////////////////////// prefetching ////////////////////////////
+
+ @Override
+ public boolean executePrefetch(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ return false;
+ }
+ boolean anyPrefetched = false;
+ for (final ConsensusPrefetchingQueue q : queues) {
+ if (!q.isClosed() && q.executePrefetch()) {
+ anyPrefetched = true;
+ }
+ }
+ return anyPrefetched;
+ }
+
+ @Override
+ public int getEventCount(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues)) {
+ return 0;
+ }
+ return queues.stream().mapToInt(ConsensusPrefetchingQueue::getPrefetchedEventCount).sum();
+ }
+
+ @Override
+ public int getQueueCount() {
+ return topicNameToConsensusPrefetchingQueues.size();
+ }
+
+ //////////////////////////// queue management ////////////////////////////
+
+ public void bindConsensusPrefetchingQueue(
+ final String topicName,
+ final String consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final long startSearchIndex) {
+ // Get or create the list of queues for this topic
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.computeIfAbsent(
+ topicName, k -> new CopyOnWriteArrayList<>());
+
+ // Check for duplicate region binding
+ for (final ConsensusPrefetchingQueue existing : queues) {
+ if (consensusGroupId.equals(existing.getConsensusGroupId()) && !existing.isClosed()) {
+ LOGGER.info(
+ "Subscription: consensus prefetching queue for topic [{}], region [{}] "
+ + "in consumer group [{}] already exists, skipping",
+ topicName,
+ consensusGroupId,
+ brokerId);
+ return;
+ }
+ }
+
+ // Get or create the shared commit ID generator for this topic
+ final AtomicLong sharedCommitIdGenerator =
+ topicNameToCommitIdGenerator.computeIfAbsent(topicName, k -> new AtomicLong(0));
+
+ final ConsensusPrefetchingQueue consensusQueue =
+ new ConsensusPrefetchingQueue(
+ brokerId,
+ topicName,
+ consensusGroupId,
+ serverImpl,
+ converter,
+ commitManager,
+ startSearchIndex,
+ sharedCommitIdGenerator);
+ queues.add(consensusQueue);
+ LOGGER.info(
+ "Subscription: create consensus prefetching queue bound to topic [{}] for consumer group [{}], "
+ + "consensusGroupId={}, startSearchIndex={}, totalRegionQueues={}",
+ topicName,
+ brokerId,
+ consensusGroupId,
+ startSearchIndex,
+ queues.size());
+ }
+
+ public void unbindConsensusPrefetchingQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
+ topicNameToConsensusPrefetchingQueues.get(topicName);
+ if (Objects.isNull(queues) || queues.isEmpty()) {
+ LOGGER.warn(
+ "Subscription: consensus prefetching queues bound to topic [{}] for consumer group [{}] do not exist",
+ topicName,
+ brokerId);
+ return;
+ }
+
+ for (final ConsensusPrefetchingQueue q : queues) {
+ q.close();
+ }
+ topicNameToConsensusPrefetchingQueues.remove(topicName);
+ topicNameToCommitIdGenerator.remove(topicName);
+ LOGGER.info(
+ "Subscription: drop all {} consensus prefetching queue(s) bound to topic [{}] for consumer group [{}]",
+ queues.size(),
+ topicName,
+ brokerId);
+ }
+
+ public int unbindByRegion(final String regionId) {
+ int closedCount = 0;
+ for (final Map.Entry<String, List<ConsensusPrefetchingQueue>> entry :
+ topicNameToConsensusPrefetchingQueues.entrySet()) {
+ final List<ConsensusPrefetchingQueue> queues = entry.getValue();
+ final Iterator<ConsensusPrefetchingQueue> iterator = queues.iterator();
+ while (iterator.hasNext()) {
+ final ConsensusPrefetchingQueue q = iterator.next();
+ if (regionId.equals(q.getConsensusGroupId())) {
+ q.close();
+ iterator.remove();
+ closedCount++;
+ LOGGER.info(
+ "Subscription: closed consensus prefetching queue for topic [{}] region [{}] "
+ + "in consumer group [{}] due to region removal",
+ entry.getKey(),
+ regionId,
+ brokerId);
+ }
+ }
+ }
+ return closedCount;
+ }
+
+ @Override
+ public void removeQueue(final String topicName) {
+ final List<ConsensusPrefetchingQueue> queues =
Review Comment:
Seemingly duplicated logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3108227868
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java:
##########
@@ -872,17 +955,41 @@ void checkAndUpdateIndex() {
}
/**
- * If there is only one replica, set it to Long.MAX_VALUE. If there are multiple replicas, get the
- * latest SafelyDeletedSearchIndex again. This enables wal to be deleted in a timely manner.
+ * Computes and updates the safe-to-delete WAL search index based on replication progress and
+ * subscription WAL retention policy. When no subscriptions exist, WAL is cleaned normally.
*/
- void checkAndUpdateSafeDeletedSearchIndex() {
+ public void checkAndUpdateSafeDeletedSearchIndex() {
if (configuration.isEmpty()) {
logger.error(
"Configuration is empty, which is unexpected. Safe deleted search index won't be updated this time.");
- } else if (configuration.size() == 1) {
+ return;
+ }
+
+ final boolean hasSubscriptions = !subscriptionQueues.isEmpty();
+ final long retentionSizeLimit =
+ config.getReplication().getSubscriptionWalRetentionSizeInBytes();
+
+ if (configuration.size() == 1 && !hasSubscriptions) {
+ // Single replica, no subscription consumers => delete all WAL freely
consensusReqReader.setSafelyDeletedSearchIndex(Long.MAX_VALUE);
} else {
- consensusReqReader.setSafelyDeletedSearchIndex(getMinFlushedSyncIndex());
+ final long replicationIndex =
+ configuration.size() > 1 ? getMinFlushedSyncIndex() : Long.MAX_VALUE;
+
+ // Subscription WAL retention: if subscriptions exist and retention is configured,
+ // prevent WAL deletion when total WAL size is within the retention limit.
+ long subscriptionRetentionBound = Long.MAX_VALUE;
+ if (hasSubscriptions && retentionSizeLimit > 0) {
+ final long totalWalSize = consensusReqReader.getTotalSize();
+ if (totalWalSize <= retentionSizeLimit) {
+ // WAL size is within retention limit — preserve all WAL for subscribers
+ subscriptionRetentionBound = ConsensusReqReader.DEFAULT_SAFELY_DELETED_SEARCH_INDEX;
Review Comment:
1. Fixed. 2. This is intentional: similar to Kafka, once data falls outside the configured retention window we do not keep it just because a consumer has not committed it yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279085372
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/io/WALMetaData.java:
##########
@@ -78,16 +125,46 @@ public void addAll(WALMetaData metaData) {
}
buffersSize.addAll(metaData.getBuffersSize());
memTablesId.addAll(metaData.getMemTablesId());
+ physicalTimes.addAll(metaData.getPhysicalTimes());
+ nodeIds.addAll(metaData.getNodeIds());
+ localSeqs.addAll(metaData.getLocalSeqs());
+ if (metaData.minDataTs < this.minDataTs) {
+ this.minDataTs = metaData.minDataTs;
+ }
+ if (metaData.maxDataTs > this.maxDataTs) {
+ this.maxDataTs = metaData.maxDataTs;
+ }
}
@Override
public int serializedSize() {
- return FIXED_SERIALIZED_SIZE
- + buffersSize.size() * Integer.BYTES
- + (memTablesId.isEmpty() ? 0 : Integer.BYTES + memTablesId.size() * Long.BYTES);
+ return serializedSize(WALFileVersion.V2);
Review Comment:
The no-version overload keeps the previous V2 behavior for compatibility. V3 is used explicitly when the WAL file version is V3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279120209
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
+ }
+
+ private static final class SearchableRequestMeta {
+ private final long searchIndex;
+ private final long physicalTime;
+ private final int nodeId;
+ private final long localSeq;
+
+ private SearchableRequestMeta(
+ final long searchIndex, final long physicalTime, final int nodeId, final long localSeq) {
+ this.searchIndex = searchIndex;
+ this.physicalTime = physicalTime;
+ this.nodeId = nodeId;
+ this.localSeq = localSeq;
+ }
+ }
+
+ private static void forEachSealedSearchableRequest(
+ final File logDir, final SearchableRequestVisitor visitor) {
+ final File[] walFiles = listSealedWALFiles(logDir);
+ if (walFiles == null || walFiles.length == 0) {
+ return;
+ }
+
+ for (final File walFile : walFiles) {
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ long pendingSearchIndex = Long.MIN_VALUE;
+ long pendingPhysicalTime = 0L;
+ int pendingNodeId = -1;
+ long pendingLocalSeq = Long.MIN_VALUE;
+ boolean hasPending = false;
+
+ while (reader.hasNext()) {
+ final ByteBuffer buffer = reader.next();
+ final WALEntryType type = WALEntryType.valueOf(buffer.get());
+ buffer.clear();
Review Comment:
The reset is needed because FileChannel.read() leaves the buffer position after the written bytes, and we need to read the long from the beginning. flip() would be equivalent and clearer here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279155322
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
Review Comment:
Changed replay lookup to scan the WAL iterator directly instead of materializing all replay requests in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3353739260
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
Review Comment:
Done in 75bacb1. Added return-value javadoc directly to `SearchableRequestVisitor.onRequest(SearchableRequestMeta request)` explaining that true continues scanning and false stops immediately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3353686592
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -731,22 +728,19 @@ public void updateCompletedTopicNames(final String consumerGroupId, final String
}
public void unbindPrefetchingQueue(final String consumerGroupId, final String topicName) {
- // Try consensus broker first
- final ConsensusSubscriptionBroker consensusBroker =
- consumerGroupIdToConsensusBroker.get(consumerGroupId);
- if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
- consensusBroker.unbindConsensusPrefetchingQueue(topicName);
- prefetchingQueueCount.invalidate();
- return;
- }
- // Fall back to pipe broker
- final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
- if (Objects.isNull(pipeBroker)) {
+ final ISubscriptionBroker broker = getBrokerForTopic(consumerGroupId, topicName);
+ if (Objects.isNull(broker)) {
LOGGER.warn(
"Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
return;
}
- pipeBroker.unbindPrefetchingQueue(topicName);
+ if (broker instanceof ConsensusSubscriptionBroker) {
+ ((ConsensusSubscriptionBroker) broker).unbindConsensusPrefetchingQueue(topicName);
+ } else if (broker instanceof SubscriptionBroker) {
+ ((SubscriptionBroker) broker).unbindPrefetchingQueue(topicName);
+ } else {
+ broker.removeQueue(topicName);
+ }
Review Comment:
Done in 75bacb1. Added `ISubscriptionBroker.unbind(String topicName)` and the agent now calls the broker interface directly.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -797,23 +786,27 @@ public int getPrefetchingQueueCount() {
public Map<String, Long> getConsensusLagSummary() {
final Map<String, Long> result = new ConcurrentHashMap<>();
- for (final Map.Entry<String, ConsensusSubscriptionBroker> entry :
- consumerGroupIdToConsensusBroker.entrySet()) {
- final String groupId = entry.getKey();
- for (final Map.Entry<String, Long> lag : entry.getValue().getLagSummary().entrySet()) {
- result.put(groupId + "/" + lag.getKey(), lag.getValue());
+ for (final Map.Entry<String, List<ISubscriptionBroker>> entry :
+ consumerGroupIdToBrokers.entrySet()) {
+ for (final ISubscriptionBroker broker : entry.getValue()) {
+ if (!(broker instanceof ConsensusSubscriptionBroker)) {
+ continue;
+ }
+ for (final Map.Entry<String, Long> lag :
+ ((ConsensusSubscriptionBroker) broker).getLagSummary().entrySet()) {
+ result.put(entry.getKey() + "/" + lag.getKey(), lag.getValue());
+ }
}
Review Comment:
Done in 75bacb1. `ISubscriptionBroker.getLagSummary()` now defaults to an empty map, so the pipe broker returns an empty summary through the interface.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -297,17 +297,25 @@ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
}
- // Build column arrays and bitmaps using bulk copy
- final long[] newTimes = Arrays.copyOf(times, rowCount);
+ // Reuse complete tablets on the server-side read path. Column filtering changes tablet shape
+ // and still needs copies for the selected columns.
+ final long[] newTimes = allColumnsMatch ? times : Arrays.copyOf(times, rowCount);
Review Comment:
Done in 75bacb1. Column filtering now reuses the selected timestamp, value, and bitmap arrays from WAL InsertNodes instead of copying column contents.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
Review Comment:
Done in 75bacb1. Added return-value javadoc to `SearchableRequestVisitor.onRequest(SearchableRequestMeta request)` explaining continue vs stop semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2921910617
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java:
##########
@@ -2477,6 +2484,52 @@ public long getSubscriptionMetaSyncerSyncIntervalMinutes() {
return subscriptionMetaSyncerSyncIntervalMinutes;
}
+ public int getSubscriptionConsensusBatchMaxDelayInMs() {
+ return subscriptionConsensusBatchMaxDelayInMs;
+ }
+
+ public void setSubscriptionConsensusBatchMaxDelayInMs(
+ final int subscriptionConsensusBatchMaxDelayInMs) {
+ this.subscriptionConsensusBatchMaxDelayInMs = subscriptionConsensusBatchMaxDelayInMs;
+ }
+
+ public long getSubscriptionConsensusBatchMaxSizeInBytes() {
+ return subscriptionConsensusBatchMaxSizeInBytes;
+ }
+
+ public void setSubscriptionConsensusBatchMaxSizeInBytes(
+ final long subscriptionConsensusBatchMaxSizeInBytes) {
+ this.subscriptionConsensusBatchMaxSizeInBytes = subscriptionConsensusBatchMaxSizeInBytes;
+ }
+
+ public int getSubscriptionConsensusBatchMaxTabletCount() {
+ return subscriptionConsensusBatchMaxTabletCount;
+ }
+
+ public void setSubscriptionConsensusBatchMaxTabletCount(
+ final int subscriptionConsensusBatchMaxTabletCount) {
+ this.subscriptionConsensusBatchMaxTabletCount = subscriptionConsensusBatchMaxTabletCount;
+ }
+
+ public int getSubscriptionConsensusBatchMaxWalEntries() {
+ return subscriptionConsensusBatchMaxWalEntries;
+ }
+
+ public void setSubscriptionConsensusBatchMaxWalEntries(
+ final int subscriptionConsensusBatchMaxWalEntries) {
+ this.subscriptionConsensusBatchMaxWalEntries = subscriptionConsensusBatchMaxWalEntries;
+ }
+
+ public long getSubscriptionConsensusWalRetentionSizeInBytes() {
+ return subscriptionConsensusWalRetentionSizeInBytes;
+ }
+
+ public void setSubscriptionConsensusWalRetentionSizeInBytes(
Review Comment:
Is it called?
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java:
##########
@@ -389,6 +389,13 @@ public class CommonConfig {
private long subscriptionMetaSyncerInitialSyncDelayMinutes = 3;
private long subscriptionMetaSyncerSyncIntervalMinutes = 3;
+ private int subscriptionConsensusBatchMaxDelayInMs = 50;
Review Comment:
May any of them support hot-reloading?
##########
iotdb-client/service-rpc/src/main/java/org/apache/iotdb/rpc/subscription/payload/request/PipeSubscribeSeekReq.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.rpc.subscription.payload.request;
+
+import org.apache.iotdb.service.rpc.thrift.TPipeSubscribeReq;
+
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Objects;
+
+public class PipeSubscribeSeekReq extends TPipeSubscribeReq {
+
+ /** Seek type constants. */
+ public static final short SEEK_TO_BEGINNING = 1;
+
+ public static final short SEEK_TO_END = 2;
+ public static final short SEEK_TO_TIMESTAMP = 3;
+
+ private transient String topicName;
+ private transient short seekType;
+ private transient long timestamp; // only meaningful when seekType == SEEK_TO_TIMESTAMP
+
+ public String getTopicName() {
+ return topicName;
+ }
+
+ public short getSeekType() {
+ return seekType;
+ }
+
+ public long getTimestamp() {
+ return timestamp;
+ }
+
+ /////////////////////////////// Thrift ///////////////////////////////
+
+ /**
+ * Serialize the incoming parameters into {@code PipeSubscribeSeekReq}, called by the subscription
+ * client.
+ */
+ public static PipeSubscribeSeekReq toTPipeSubscribeReq(
+ final String topicName, final short seekType, final long timestamp)
+ throws IOException {
+ final PipeSubscribeSeekReq req = new PipeSubscribeSeekReq();
+
+ req.topicName = topicName;
+ req.seekType = seekType;
+ req.timestamp = timestamp;
+
+ req.version = PipeSubscribeRequestVersion.VERSION_1.getVersion();
+ req.type = PipeSubscribeRequestType.SEEK.getType();
+ try (final PublicBAOS byteArrayOutputStream = new PublicBAOS();
+ final DataOutputStream outputStream = new DataOutputStream(byteArrayOutputStream)) {
+ ReadWriteIOUtils.write(topicName, outputStream);
+ ReadWriteIOUtils.write(seekType, outputStream);
+ if (seekType == SEEK_TO_TIMESTAMP) {
+ ReadWriteIOUtils.write(timestamp, outputStream);
+ }
+ req.body = ByteBuffer.wrap(byteArrayOutputStream.getBuf(), 0, byteArrayOutputStream.size());
+ }
+
+ return req;
+ }
+
+ /** Deserialize {@code TPipeSubscribeReq} to obtain parameters, called by the subscription server. */
+ public static PipeSubscribeSeekReq fromTPipeSubscribeReq(final TPipeSubscribeReq seekReq) {
+ final PipeSubscribeSeekReq req = new PipeSubscribeSeekReq();
+
+ if (Objects.nonNull(seekReq.body) && seekReq.body.hasRemaining()) {
+ req.topicName = ReadWriteIOUtils.readString(seekReq.body);
+ req.seekType = ReadWriteIOUtils.readShort(seekReq.body);
+ if (req.seekType == SEEK_TO_TIMESTAMP) {
+ req.timestamp = ReadWriteIOUtils.readLong(seekReq.body);
+ }
+ }
+
+ req.version = seekReq.version;
+ req.type = seekReq.type;
+ req.body = seekReq.body;
Review Comment:
May delete this to help GC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3278708133
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java:
##########
@@ -720,13 +807,125 @@ public IndexedConsensusRequest buildIndexedConsensusRequestForLocalRequest(
new IoTProgressIndex(thisNode.getNodeId(), searchIndex.get() + 1);
((ComparableConsensusRequest) request).setProgressIndex(iotProgressIndex);
}
- return new IndexedConsensusRequest(searchIndex.get() + 1, Collections.singletonList(request));
+ return new IndexedConsensusRequest(searchIndex.get() + 1, Collections.singletonList(request))
+ .setPhysicalTime(assignPhysicalTimeInMs())
+ .setNodeId(thisNode.getNodeId());
}
public IndexedConsensusRequest buildIndexedConsensusRequestForRemoteRequest(
- long syncIndex, List<IConsensusRequest> requests) {
- return new IndexedConsensusRequest(
- ConsensusReqReader.DEFAULT_SEARCH_INDEX, syncIndex, requests);
+ long syncIndex,
+ long routingEpoch,
+ long physicalTime,
+ int nodeId,
+ List<IConsensusRequest> requests) {
+ observePhysicalTimeLowerBound(physicalTime);
+ IndexedConsensusRequest req =
+ new IndexedConsensusRequest(ConsensusReqReader.DEFAULT_SEARCH_INDEX, syncIndex, requests);
+ req.setRoutingEpoch(routingEpoch);
+ req.setPhysicalTime(physicalTime);
+ req.setNodeId(nodeId);
+ return req;
+ }
+
+ public WriterSafeFrontierTracker.SafeHlc createIdleSafeHlcForCurrentWriter() {
+ final long safePhysicalTime = assignPhysicalTimeInMs();
+ final long barrierLocalSeq = searchIndex.get();
+ writerSafeFrontierTracker.recordAppliedProgress(
+ safePhysicalTime, thisNode.getNodeId(), barrierLocalSeq);
+ return new WriterSafeFrontierTracker.SafeHlc(safePhysicalTime, barrierLocalSeq);
+ }
+
+ public void observeRemoteSafeHlc(
+ final long safePhysicalTime, final int writerNodeId, final long barrierLocalSeq) {
+ observePhysicalTimeLowerBound(safePhysicalTime);
+ writerSafeFrontierTracker.observePendingSafeHlc(
+ safePhysicalTime, writerNodeId, barrierLocalSeq);
+ }
+
+ public void recordRemoteAppliedWriterProgress(
+ final long physicalTime, final int writerNodeId, final long appliedLocalSeq) {
+ writerSafeFrontierTracker.recordAppliedProgress(physicalTime, writerNodeId, appliedLocalSeq);
+ }
+
+ public long getEffectiveSafePhysicalTime(final int writerNodeId) {
+ return writerSafeFrontierTracker.getEffectiveSafePt(writerNodeId);
+ }
+
+ public WriterSafeFrontierTracker getWriterSafeFrontierTracker() {
+ return writerSafeFrontierTracker;
+ }
+
+ public boolean hasSubscriptionConsumers() {
+ return !subscriptionQueueRegistry.isEmpty();
+ }
+
+ private long assignPhysicalTimeInMs() {
+ while (true) {
+ final long previous = lastAssignedPhysicalTime.get();
+ final long candidate = Math.max(System.currentTimeMillis(), previous);
+ if (lastAssignedPhysicalTime.compareAndSet(previous, candidate)) {
+ return candidate;
+ }
+ }
+ }
+
+ private void observePhysicalTimeLowerBound(final long observedPhysicalTime) {
+ if (observedPhysicalTime <= 0) {
+ return;
+ }
+ while (true) {
+ final long previous = lastAssignedPhysicalTime.get();
+ final long candidate = Math.max(previous, observedPhysicalTime);
+ if (candidate == previous || lastAssignedPhysicalTime.compareAndSet(previous, candidate)) {
+ return;
+ }
+ }
+ }
+
+ private void initializeWriterMeta() {
+ final long recoveredSearchIndex = searchIndex.get();
+ try {
+ final Optional<WriterMeta> writerMetaOptional = WriterMeta.load(writerMetaPath);
+ if (writerMetaOptional.isPresent()) {
+ final WriterMeta writerMeta = writerMetaOptional.get();
+ logger.info(
+ "Recovered writer meta for group {} from {}, recoveredLocalSeq={}, "
+ + "persistedLocalSeq={}",
+ consensusGroupId,
+ writerMetaPath,
+ recoveredSearchIndex,
+ writerMeta.getLastAllocatedLocalSeq());
+ lastAssignedPhysicalTime.set(
+ Math.max(writerMeta.getLastAssignedPhysicalTimeMs(), System.currentTimeMillis()));
+ return;
+ }
+ } catch (IOException e) {
+ logger.warn(
+ "Failed to load writer meta for group {} from {}. Starting fresh writer metadata.",
+ consensusGroupId,
+ writerMetaPath,
+ e);
+ }
+ lastAssignedPhysicalTime.set(System.currentTimeMillis());
+ logger.info(
+ "Initialized fresh writer meta for group {}, recoveredLocalSeq={}",
+ consensusGroupId,
+ recoveredSearchIndex);
+ }
+
+ private void persistWriterMetaOnSuccess(final IndexedConsensusRequest indexedConsensusRequest) {
+ try {
+ new WriterMeta(
+ indexedConsensusRequest.getLocalSeq(), indexedConsensusRequest.getPhysicalTime())
+ .persist(writerMetaPath);
+ } catch (IOException e) {
+ logger.warn(
+ "Failed to persist writer meta for group {} at localSeq={}, pt={}",
+ consensusGroupId,
+ indexedConsensusRequest.getLocalSeq(),
+ indexedConsensusRequest.getPhysicalTime(),
+ e);
+ }
}
Review Comment:
Avoided persisting on every successful write. The latest value is kept in memory, persisted at most once per second, and flushed on stop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279126000
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
+ }
+
+ private static final class SearchableRequestMeta {
+ private final long searchIndex;
+ private final long physicalTime;
+ private final int nodeId;
+ private final long localSeq;
+
+ private SearchableRequestMeta(
+ final long searchIndex, final long physicalTime, final int nodeId, final long localSeq) {
+ this.searchIndex = searchIndex;
+ this.physicalTime = physicalTime;
+ this.nodeId = nodeId;
+ this.localSeq = localSeq;
+ }
+ }
+
+ private static void forEachSealedSearchableRequest(
+ final File logDir, final SearchableRequestVisitor visitor) {
+ final File[] walFiles = listSealedWALFiles(logDir);
+ if (walFiles == null || walFiles.length == 0) {
+ return;
+ }
+
+ for (final File walFile : walFiles) {
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ long pendingSearchIndex = Long.MIN_VALUE;
+ long pendingPhysicalTime = 0L;
+ int pendingNodeId = -1;
+ long pendingLocalSeq = Long.MIN_VALUE;
+ boolean hasPending = false;
+
+ while (reader.hasNext()) {
+ final ByteBuffer buffer = reader.next();
Review Comment:
Now explicitly uses WAL footer metadata and does not deserialize entry bodies for this search path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279178062
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = inFlightEvents.get(new Pair<>(consumerId, commitContext));
+ if (Objects.isNull(event)) {
+ if (isCommitContextOutdated(commitContext)) {
+ return generateOutdatedErrorResponse();
+ }
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
+ this, consumerId, commitContext));
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ // ======================== Prefetch Round Drive ========================
+
+ private static final long WAL_GAP_RETRY_SLEEP_MS = 10L;
+ private static final long WAL_GAP_WAIT_LOG_INTERVAL_MS = 5_000L;
+
+ private static final long PREFETCH_STATS_LOG_INTERVAL_MS = 5_000L;
+
+ public PrefetchRoundResult drivePrefetchOnce() {
+ if (applyPendingSeekRequestIfNecessary()) {
+ return closeRequested ? PrefetchRoundResult.dormant() : PrefetchRoundResult.rescheduleNow();
+ }
+
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !prefetchInitialized) {
+ return PrefetchRoundResult.dormant();
+ }
+
+ logPeriodicStatsIfNecessary();
+
+ final long currentSeekGeneration = seekGeneration.get();
+ if (currentSeekGeneration != observedSeekGeneration) {
+ resetRoundStateForSeek(currentSeekGeneration);
+ }
+
+ applyPendingSubscriptionWalReset(observedSeekGeneration);
+ recycleInFlightEvents();
+
+ if (!isActive || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return computeIdleRoundResult();
+ }
+
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+ final int batchMaxDelayMs = config.getSubscriptionConsensusBatchMaxDelayInMs();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+
+ final List<IndexedConsensusRequest> batch = drainPendingEntries(maxWalEntries);
+ if (!batch.isEmpty()) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: drained {} entries from pendingEntries, "
+ + "first searchIndex={}, last searchIndex={}, nextExpected={}, "
+ + "prefetchingQueueSize={}",
+ this,
+ batch.size(),
+ batch.get(0).getSearchIndex(),
+ batch.get(batch.size() - 1).getSearchIndex(),
+ nextExpectedSearchIndex.get(),
+ prefetchingQueue.size());
+
+ final boolean batchAccepted =
+ accumulateFromPending(
+ batch, lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes);
+ if (!batchAccepted) {
+ if (pendingWalGapRetryRequested) {
+ // Once a drained batch hits an unresolved WAL gap, the affected suffix falls back to
+ // the WAL path on later rounds instead of being requeued into the bounded pending path.
+ return PrefetchRoundResult.rescheduleAfter(WAL_GAP_RETRY_SLEEP_MS);
+ }
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ }
+
+ if (batch.isEmpty() && lingerBatch.isEmpty()) {
+ tryCatchUpFromWAL(observedSeekGeneration);
+ }
+
+ if (!drainBufferedRealtimeLanes(
+ lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes)) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerElapsedMs = System.currentTimeMillis() - lingerBatch.firstTabletTimeMs;
+ if (lingerElapsedMs >= batchMaxDelayMs) {
+ if (seekGeneration.get() != observedSeekGeneration) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: time-based flush, {} tablets lingered for {}ms "
+ + "(threshold={}ms)",
+ this,
+ lingerBatch.tablets.size(),
+ lingerElapsedMs,
+ batchMaxDelayMs);
+ flushBatch(lingerBatch, observedSeekGeneration);
+ }
+ }
+
+ maybeInjectWatermark();
+ return computeIdleRoundResult();
+ } catch (final Throwable fatal) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: prefetch round failed " + "(type={}, message={})",
+ this,
+ fatal.getClass().getName(),
+ fatal.getMessage(),
+ fatal);
+ if (fatal instanceof VirtualMachineError) {
+ markClosed();
+ return PrefetchRoundResult.dormant();
+ }
+ return PrefetchRoundResult.rescheduleAfter(100L);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private void logPeriodicStatsIfNecessary() {
+ final long nowMs = System.currentTimeMillis();
+ if (nowMs - lastStatsLogTimeMs < PREFETCH_STATS_LOG_INTERVAL_MS) {
+ return;
+ }
+
+ final long currentPendingAcceptedEntries = pendingPathAcceptedEntries.get();
+ final long currentWalAcceptedEntries = walPathAcceptedEntries.get();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: periodic stats, lag={}, pendingDelta={}, walDelta={}, "
+ + "pendingTotal={}, walTotal={}, pendingQueueSize={}, prefetchingQueueSize={}, "
+ + "inFlightEventsSize={}, realtimeLaneCount={}, walHasNext={}, isActive={}, subtaskScheduled={}",
+ this,
+ getLag(),
+ currentPendingAcceptedEntries - lastPendingAcceptedEntries,
+ currentWalAcceptedEntries - lastWalAcceptedEntries,
+ currentPendingAcceptedEntries,
+ currentWalAcceptedEntries,
+ pendingEntries.size(),
+ prefetchingQueue.size(),
+ inFlightEvents.size(),
+ realtimeEntriesByLane.size(),
+ hasReadableWalEntries(),
+ isActive,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ lastStatsLogTimeMs = nowMs;
+ lastPendingAcceptedEntries = currentPendingAcceptedEntries;
+ lastWalAcceptedEntries = currentWalAcceptedEntries;
+ }
+
+ private void resetRoundStateForSeek(final long newSeekGeneration) {
+ restorePendingSubscriptionWalCursor(newSeekGeneration);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = newSeekGeneration;
+ }
+
+ private List<IndexedConsensusRequest> drainPendingEntries(final int maxWalEntries) {
+ final List<IndexedConsensusRequest> batch = new ArrayList<>();
+ IndexedConsensusRequest next;
+ while (batch.size() < maxWalEntries && (next = pendingEntries.poll()) != null) {
+ batch.add(next);
+ }
+ return batch;
+ }
+
+ private PrefetchRoundResult computeIdleRoundResult() {
+ if (isClosed || !prefetchInitialized || !isActive) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (hasImmediatePrefetchableWork()) {
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ long delayMs = Long.MAX_VALUE;
+ if (hasHistoricalWalLag()) {
+ delayMs = Math.min(delayMs, WAL_GAP_RETRY_SLEEP_MS);
+ }
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerDelayMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusBatchMaxDelayInMs()
+ - (System.currentTimeMillis() - lingerBatch.firstTabletTimeMs);
+ delayMs = Math.min(delayMs, Math.max(1L, lingerDelayMs));
+ }
+
+ final long watermarkDelayMs = computeWatermarkDelayMs();
+ if (watermarkDelayMs > 0L) {
+ delayMs = Math.min(delayMs, watermarkDelayMs);
+ }
+
+ if (!inFlightEvents.isEmpty()) {
+ delayMs =
+ Math.min(
+ delayMs,
+ SubscriptionConfig.getInstance().getSubscriptionRecycleUncommittedEventIntervalMs());
+ }
+
+ return delayMs == Long.MAX_VALUE
+ ? PrefetchRoundResult.dormant()
+ : PrefetchRoundResult.rescheduleAfter(delayMs);
+ }
+
+ private long computeWatermarkDelayMs() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return -1L;
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0L) {
+ return -1L;
+ }
+ if (lastWatermarkEmitTimeMs == 0L) {
+ return 1L;
+ }
+ final long elapsedMs = System.currentTimeMillis() - lastWatermarkEmitTimeMs;
+ return elapsedMs >= intervalMs ? 1L : Math.max(1L, intervalMs - elapsedMs);
+ }
+
+ private boolean hasImmediatePrefetchableWork() {
+ return !pendingEntries.isEmpty() || !realtimeEntriesByLane.isEmpty() || hasReadableWalEntries();
+ }
+
+ private boolean hasHistoricalWalLag() {
+ return nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex();
+ }
+
+ /**
+ * Accumulates tablets from pending entries into the linger buffer. When pending replay outruns
+ * the local WAL reader, this method backfills the local-index gap from WAL before continuing.
+ *
+ * @return false if the batch became stale because seek generation changed while flushing
+ */
+ private static boolean hasLocalSearchIndex(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() >= 0;
+ }
+
+ private boolean isBeforeLocalCursor(final IndexedConsensusRequest request) {
+ return hasLocalSearchIndex(request) && request.getSearchIndex() < nextExpectedSearchIndex.get();
+ }
+
+ private void advanceLocalCursorIfPresent(final IndexedConsensusRequest request) {
+ if (hasLocalSearchIndex(request)) {
+ nextExpectedSearchIndex.set(request.getSearchIndex() + 1);
+ }
+ }
+
+ private boolean appendRealtimeRequest(
+ final IndexedConsensusRequest request,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean fromPending) {
+ final PreparedEntry preparedEntry = prepareEntry(request);
+ if (Objects.isNull(preparedEntry)) {
+ return true;
+ }
+ if (!appendPreparedEntryViaRealtimeLane(
+ batchState, preparedEntry, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+ if (fromPending) {
+ markAcceptedFromPending();
+ } else {
+ markAcceptedFromWal();
+ }
+ return true;
+ }
+
+ private boolean accumulateFromPending(
+ final List<IndexedConsensusRequest> batch,
+ final DeliveryBatchState lingerBatch,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+
+ int processedCount = 0;
+ int skippedCount = 0;
+
+ for (int index = 0; index < batch.size(); index++) {
+ final IndexedConsensusRequest request = batch.get(index);
+ final long searchIndex = request.getSearchIndex();
+
+ // Only local-indexed requests participate in the internal WAL read cursor.
+ final long expected = nextExpectedSearchIndex.get();
+ if (hasLocalSearchIndex(request) && searchIndex > expected) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: gap detected, expected={}, got={}. "
+ + "Filling {} entries from WAL.",
+ this,
+ expected,
+ searchIndex,
+ searchIndex - expected);
+ if (!fillGapFromWAL(
+ expected,
+ searchIndex,
+ lingerBatch,
+ expectedSeekGeneration,
+ maxTablets,
+ maxBatchBytes)) {
+ return false;
+ }
+ }
+
+ if (isBeforeLocalCursor(request)) {
+ skippedCount++;
+ continue;
+ }
+
+ if (shouldSkipForRecoveryProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ request, lingerBatch, expectedSeekGeneration, maxTablets, maxBatchBytes, true)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(request);
+ processedCount++;
+ advanceLocalCursorIfPresent(request);
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: accumulate complete, batchSize={}, processed={}, "
+ + "skipped={}, lingerTablets={}, nextExpected={}",
+ this,
+ batch.size(),
+ processedCount,
+ skippedCount,
+ lingerBatch.tablets.size(),
+ nextExpectedSearchIndex.get());
+
+ return true;
+ }
+
+ /**
+ * Fills a gap in the pending queue by reading entries from WAL so the internal local replay
+ * cursor stays contiguous even when pending delivery jumps ahead of the WAL iterator.
+ *
+ * <p>Temporary WAL visibility lag is treated as a normal back-pressure condition: once a drained
+ * pending batch encounters an unresolved local-index gap, the queue backs off and lets the
+ * affected suffix fall back to the WAL path on later rounds. This keeps replay contiguous without
+ * requeueing the drained batch back into the bounded pending queue.
+ *
+ * @return false if gap fill had to stop because the current batch became stale or the queue was
+ * interrupted/closed
+ */
+ private boolean fillGapFromWAL(
+ final long fromIndex,
+ final long toIndex,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ pendingWalGapRetryRequested = false;
+ resetSubscriptionWALPosition(fromIndex);
+ if (seekGeneration.get() != expectedSeekGeneration || isClosed) {
+ return false;
+ }
+ if (!pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, Integer.MAX_VALUE, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final long nextExpected = nextExpectedSearchIndex.get();
+ if (nextExpected >= toIndex) {
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ return true;
+ }
+
+ final long nowMs = System.currentTimeMillis();
+ if (walGapWaitStartTimeMs == 0L) {
+ walGapWaitStartTimeMs = nowMs;
+ }
+ if (lastWalGapWaitLogTimeMs == 0L
+ || nowMs - lastWalGapWaitLogTimeMs >= WAL_GAP_WAIT_LOG_INTERVAL_MS) {
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: waiting {}ms for WAL gap [{}, {}) to become visible, "
+ + "currentNextExpected={}, currentWalIndex={}, seekGeneration={}",
+ this,
+ nowMs - walGapWaitStartTimeMs,
+ nextExpected,
+ toIndex,
+ nextExpected,
+ consensusReqReader.getCurrentSearchIndex(),
+ expectedSeekGeneration);
+ lastWalGapWaitLogTimeMs = nowMs;
+ }
+ onWalGapRetryScheduled();
+ pendingWalGapRetryRequested = true;
+ return false;
+ }
+
+ /**
+ * Try catch-up from WAL when the pending queue was empty. This handles cold-start or scenarios
+ * where the subscription started after data was already written.
+ */
+ private void tryCatchUpFromWAL(final long expectedSeekGeneration) {
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+
+ final DeliveryBatchState batchState = new DeliveryBatchState();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ final boolean accepted =
+ pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, maxWalEntries, maxTablets, maxBatchBytes);
+ if (!accepted) {
+ return;
+ }
+
+ if (!batchState.isEmpty()) {
+ flushBatch(batchState, expectedSeekGeneration);
+ }
+ }
+
+ private boolean pumpFromSubscriptionWAL(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxWalEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return true;
+ }
+
+ subscriptionWALIterator.refresh();
+ ensureSubscriptionWalReadable();
+
+ int entriesRead = 0;
+ while (entriesRead < maxWalEntries
+ && subscriptionWALIterator.hasNext()
+ && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ try {
+ final IndexedConsensusRequest walEntry = subscriptionWALIterator.next();
+ entriesRead++;
+
+ if (isBeforeLocalCursor(walEntry)) {
+ continue;
+ }
+ if (shouldSkipForRecoveryProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ walEntry, batchState, expectedSeekGeneration, maxTablets, maxBatchBytes, false)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(walEntry);
+ advanceLocalCursorIfPresent(walEntry);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error reading subscription WAL", this, e);
+ break;
+ }
+ }
+
+ if (entriesRead > 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL read {} entries, nextExpectedSearchIndex={}",
+ this,
+ entriesRead,
+ nextExpectedSearchIndex.get());
+ }
+ return true;
+ }
+
+ private void ensureSubscriptionWalReadable() {
+ if (Objects.isNull(subscriptionWALIterator)
+ || subscriptionWALIterator.hasNext()
+ || !(consensusReqReader instanceof WALNode)) {
+ return;
+ }
+
+ final long currentWalIndex = consensusReqReader.getCurrentSearchIndex();
+ if (nextExpectedSearchIndex.get() > currentWalIndex) {
+ return;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL exhausted at {} while current WAL is {}. "
+ + "Rolling WAL file to expose current-file entries.",
+ this,
+ nextExpectedSearchIndex.get(),
+ currentWalIndex);
+ ((WALNode) consensusReqReader).rollWALFile();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ if (Objects.nonNull(subscriptionWALIterator)) {
+ subscriptionWALIterator.refresh();
+ }
+ }
+
+ private void resetSubscriptionWALPosition(final long startSearchIndex) {
+ closeSubscriptionWALIterator();
+ subscriptionWALIterator = createSubscriptionWALIterator(startSearchIndex);
+ }
+
+ protected ProgressWALIterator createSubscriptionWALIterator(final long startSearchIndex) {
+ if (consensusReqReader instanceof WALNode) {
+ return new ProgressWALIterator((WALNode) consensusReqReader, startSearchIndex);
+ }
+ return null;
+ }
+
+ protected void onWalGapRetryScheduled() {}
+
+ private boolean hasReadableWalEntries() {
+ return Objects.nonNull(subscriptionWALIterator) && subscriptionWALIterator.hasNext();
+ }
+
+ private void requestSubscriptionWalReset(
+ final long targetSearchIndex, final long seekGenerationValue) {
+ pendingSubscriptionWalResetSearchIndex = targetSearchIndex;
+ pendingSubscriptionWalResetGeneration = seekGenerationValue;
+ }
+
+ private void applyPendingSubscriptionWalReset(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ resetSubscriptionWALPosition(pendingSubscriptionWalResetSearchIndex);
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ }
+
+ private void restorePendingSubscriptionWalCursor(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ // A seek can land in the middle of a prefetch iteration. Restore the local cursor to the
+ // pending seek target before resuming under the new generation so stale in-flight work does
+ // not permanently advance the historical replay frontier.
+ nextExpectedSearchIndex.set(pendingSubscriptionWalResetSearchIndex);
+ }
+
+ private void closeSubscriptionWALIterator() {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return;
+ }
+ try {
+ subscriptionWALIterator.close();
+ } catch (final IOException e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error closing subscription WAL iterator", this, e);
+ } finally {
+ subscriptionWALIterator = null;
+ }
+ }
+
+ /**
+ * Deserializes the IConsensusRequest entries within an IndexedConsensusRequest to produce an
+ * InsertNode. WAL entries are typically stored as IoTConsensusRequest (serialized ByteBuffers),
+ * and a single logical write may be split across multiple fragments (SearchNode). This method
+ * handles both cases.
+ *
+ * <p>The deserialization follows the same pattern as {@code
+ * DataRegionStateMachine.grabPlanNode()}.
+ */
+ private InsertNode deserializeToInsertNode(final IndexedConsensusRequest indexedRequest) {
+ final List<SearchNode> searchNodes = new ArrayList<>();
+ PlanNode nonSearchNode = null;
+
+ for (final IConsensusRequest req : indexedRequest.getRequests()) {
+ PlanNode planNode;
+ try {
+ if (req instanceof IoTConsensusRequest) {
+ // WAL entries read from file are wrapped as IoTConsensusRequest (ByteBuffer)
+ planNode = WALEntry.deserializeForConsensus(req.serializeToByteBuffer());
+ } else if (req instanceof InsertNode) {
+ // In-memory entries (not yet flushed to WAL file) may already be PlanNode
+ planNode = (PlanNode) req;
+ } else {
+ // ByteBufferConsensusRequest or unknown
+ planNode = PlanNodeType.deserialize(req.serializeToByteBuffer());
+ }
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to deserialize IConsensusRequest "
+ + "(type={}) in searchIndex={}: {}",
+ this,
+ req.getClass().getSimpleName(),
+ indexedRequest.getSearchIndex(),
+ e.getMessage(),
+ e);
+ continue;
+ }
+
+ if (planNode instanceof SearchNode) {
+ final SearchNode searchNode = (SearchNode) planNode;
+ searchNode.setSearchIndex(indexedRequest.getSearchIndex());
+ if (indexedRequest.getSyncIndex() >= 0) {
+ searchNode.setSyncIndex(indexedRequest.getSyncIndex());
+ }
+ if (indexedRequest.getPhysicalTime() > 0) {
+ searchNode.setPhysicalTime(indexedRequest.getPhysicalTime());
+ }
+ if (indexedRequest.getNodeId() >= 0) {
+ searchNode.setNodeId(indexedRequest.getNodeId());
+ }
+ searchNodes.add(searchNode);
+ } else {
+ nonSearchNode = planNode;
+ }
+ }
+
+ // Merge split SearchNode fragments (same pattern as DataRegionStateMachine.grabPlanNode)
+ if (!searchNodes.isEmpty()) {
+ final PlanNode merged = searchNodes.get(0).merge(searchNodes);
+ if (merged instanceof InsertNode) {
+ final InsertNode mergedInsert = (InsertNode) merged;
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: deserialized merged InsertNode for searchIndex={}, "
+ + "type={}, deviceId={}, searchNodeCount={}",
+ this,
+ indexedRequest.getSearchIndex(),
+ mergedInsert.getType(),
+ ConsensusLogToTabletConverter.safeDeviceIdForLog(mergedInsert),
+ searchNodes.size());
+
+ return mergedInsert;
+ }
+ }
+
+ if (nonSearchNode != null) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: searchIndex={} contains non-InsertNode PlanNode: {}",
+ this,
+ indexedRequest.getSearchIndex(),
+ nonSearchNode.getClass().getSimpleName());
+ }
+
+ return null;
+ }
+
+ private PreparedEntry prepareEntry(final IndexedConsensusRequest indexedRequest) {
+ final InsertNode insertNode = deserializeToInsertNode(indexedRequest);
+ if (Objects.isNull(insertNode)) {
+ return null;
+ }
+
+ final long localSeq =
+ indexedRequest.getProgressLocalSeq() >= 0
+ ? indexedRequest.getProgressLocalSeq()
+ : indexedRequest.getSearchIndex();
+ final long searchIndex = indexedRequest.getSearchIndex();
+ final long physicalTime =
+ indexedRequest.getPhysicalTime() > 0
+ ? indexedRequest.getPhysicalTime()
+ : insertNode.getPhysicalTime();
+ final int writerNodeId =
+ indexedRequest.getNodeId() >= 0 ? indexedRequest.getNodeId() : insertNode.getNodeId();
+
+ trackWriterLane(writerNodeId);
+ final long maxTs = extractMaxTime(insertNode);
+ if (maxTs > maxObservedTimestamp) {
+ maxObservedTimestamp = maxTs;
+ }
+ final List<Tablet> tablets = converter.convert(insertNode);
+ if (tablets.isEmpty()) {
+ return null;
+ }
+
+ return new PreparedEntry(tablets, physicalTime, writerNodeId, localSeq, searchIndex);
+ }
+
+ private static long estimateTabletSize(final Tablet tablet) {
+ return PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet);
+ }
+
+ private void createAndEnqueueEvent(
+ final List<Tablet> tablets, final long startSearchIndex, final long endSearchIndex) {
+ createAndEnqueueEvent(
+ tablets, startSearchIndex, endSearchIndex, endSearchIndex, seekGeneration.get());
+ }
+
+ private boolean createAndEnqueueEvent(
+ final List<Tablet> tablets,
+ final long startSearchIndex,
+ final long endSearchIndex,
+ final long commitLocalSeq,
+ final long expectedSeekGeneration) {
+ if (tablets.isEmpty()) {
+ return true;
+ }
+
+ if (seekGeneration.get() != expectedSeekGeneration) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: skip stale event with searchIndex range [{}, {}], "
+ + "expectedSeekGeneration={}, currentSeekGeneration={}",
+ this,
+ startSearchIndex,
+ endSearchIndex,
+ expectedSeekGeneration,
+ seekGeneration.get());
+ return false;
+ }
+
+ final SubscriptionCommitContext commitContext = buildWriterCommitContext(commitLocalSeq);
+ final WriterId writerId = commitContext.getWriterId();
+ final WriterProgress writerProgress = commitContext.getWriterProgress();
+ commitManager.recordMapping(brokerId, topicName, consensusGroupId, writerId, writerProgress);
+
+ // nextOffset <= 0 means all tablets delivered in single batch
+ // -tablets.size() indicates total count
+ // Use Map<String, List<Tablet>> constructor with actual database name for table model;
+ final TabletsPayload payload =
+ new TabletsPayload(
+ Collections.singletonMap(converter.getDatabaseName(), tablets), -tablets.size());
+
+ final SubscriptionEvent event =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.TABLETS.getType(), payload, commitContext);
+
+ prefetchingQueue.add(event);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: ENQUEUED event with {} tablets, "
+ + "searchIndex range [{}, {}], prefetchQueueSize={}",
+ this,
+ tablets.size(),
+ startSearchIndex,
+ endSearchIndex,
+ prefetchingQueue.size());
+
+ // After enqueuing the data event, control metadata is handled separately from user data.
+ return true;
+ }
+
+ private SubscriptionCommitContext buildWriterCommitContext(final long localSeq) {
+ final int effectiveNodeId =
+ batchWriterNodeId >= 0
+ ? batchWriterNodeId
+ : IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), effectiveNodeId);
+ final WriterProgress writerProgress = new WriterProgress(batchPhysicalTime, localSeq);
+ return new SubscriptionCommitContext(
+ IoTDBDescriptor.getInstance().getConfig().getDataNodeId(),
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ seekGeneration.get(),
+ writerId,
+ writerProgress);
+ }
+
+ private void updateBatchWriterProgress(final long physicalTime, final int writerNodeId) {
+ if (physicalTime > 0) {
+ this.batchPhysicalTime = physicalTime;
+ }
+ if (writerNodeId >= 0) {
+ this.batchWriterNodeId = writerNodeId;
+ }
+ }
+
+ private void resetBatchWriterProgress() {
+ this.batchPhysicalTime = 0L;
+ this.batchWriterNodeId = -1;
+ }
+
+ private long estimateTabletsBytes(final List<Tablet> tablets) {
+ long estimatedBytes = 0L;
+ for (final Tablet tablet : tablets) {
+ estimatedBytes += estimateTabletSize(tablet);
+ }
+ return estimatedBytes;
+ }
+
+ private boolean appendPreparedEntryViaRealtimeLane(
+ final DeliveryBatchState batchState,
+ final PreparedEntry preparedEntry,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ bufferRealtimeEntry(preparedEntry);
+ return drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes);
+ }
+
+ private int getRealtimeBufferedEntryCount() {
+ int count = 0;
+ for (final NavigableMap<Long, PreparedEntry> laneEntries : realtimeEntriesByLane.values()) {
+ count += laneEntries.size();
+ }
+ return count;
+ }
+
+ private boolean drainBufferedRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ while (!realtimeEntriesByLane.isEmpty()) {
+ final int bufferedBefore = getRealtimeBufferedEntryCount();
+ if (!drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final int bufferedAfter = getRealtimeBufferedEntryCount();
+ if (bufferedAfter == 0 || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return true;
+ }
+
+ if (batchState.isEmpty()) {
+ return true;
+ }
+
+ if (!flushBatch(batchState, expectedSeekGeneration)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private boolean canAppendLaneEntry(
+ final DeliveryBatchState batchState,
+ final LaneBufferedEntry entry,
+ final long entryEstimatedBytes,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ final boolean wouldExceedEntryLimit =
+ maxEntries != Integer.MAX_VALUE && batchState.entryCount >= maxEntries;
+ final boolean wouldExceedTabletLimit =
+ !batchState.isEmpty() && batchState.tablets.size() + entry.getTablets().size() > maxTablets;
+ final boolean wouldExceedByteLimit =
+ !batchState.isEmpty() && batchState.estimatedBytes + entryEstimatedBytes > maxBatchBytes;
+ // Keep all consensus subscription modes on a single-writer commit/delivery shape so
+ // SubscriptionCommitContext and RegionProgress remain per-writer.
+ final boolean writerChanged =
+ !batchState.isEmpty() && batchState.writerNodeId != entry.getWriterNodeId();
+ return !(wouldExceedEntryLimit
+ || wouldExceedTabletLimit
+ || wouldExceedByteLimit
+ || writerChanged);
+ }
+
+ private boolean drainRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ return drainLaneEntries(
+ batchState,
+ this::buildRealtimeLaneFrontiers,
+ this::peekRealtimeEntry,
+ entry -> true,
+ (laneId, entry) -> removeRealtimeEntry(laneId, entry.localSeq),
+ Integer.MAX_VALUE,
+ maxTablets,
+ maxBatchBytes,
+ true);
+ }
+
+ private <T extends LaneBufferedEntry> boolean drainLaneEntries(
+ final DeliveryBatchState batchState,
+ final Supplier<PriorityQueue<LaneFrontier>> frontierSupplier,
+ final Function<WriterLaneId, T> headSupplier,
+ final Predicate<T> releasePredicate,
+ final BiConsumer<WriterLaneId, T> removeHeadAction,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean trackLingerTime) {
+ while (true) {
+ final PriorityQueue<LaneFrontier> frontiers = frontierSupplier.get();
+ if (frontiers.isEmpty()) {
+ return true;
+ }
+ final LaneFrontier frontier = frontiers.peek();
+ if (Objects.isNull(frontier) || frontier.isBarrier) {
+ return true;
+ }
+ final T laneHead = headSupplier.apply(frontier.laneId);
+ if (Objects.isNull(laneHead)) {
+ return true;
+ }
+ if (!releasePredicate.test(laneHead)) {
+ return true;
+ }
+
+ final long entryEstimatedBytes = estimateTabletsBytes(laneHead.getTablets());
+ if (!canAppendLaneEntry(
+ batchState, laneHead, entryEstimatedBytes, maxEntries, maxTablets, maxBatchBytes)) {
+ return true;
+ }
+
+ removeHeadAction.accept(frontier.laneId, laneHead);
+ batchState.append(laneHead, entryEstimatedBytes, trackLingerTime);
+ }
+ }
+
+ private boolean flushBatch(
+ final DeliveryBatchState batchState, final long expectedSeekGeneration) {
+ updateBatchWriterProgress(batchState.physicalTime, batchState.writerNodeId);
+ if (!createAndEnqueueEvent(
+ new ArrayList<>(batchState.tablets),
+ batchState.startSearchIndex,
+ batchState.endSearchIndex,
+ batchState.lastLocalSeq,
+ expectedSeekGeneration)) {
+ return false;
+ }
+ resetBatchWriterProgress();
+ batchState.reset();
+ return true;
+ }
+
+ // ======================== Commit (Ack/Nack) ========================
+
+ private boolean canAcceptCommitContext(
+ final SubscriptionCommitContext commitContext, final String action, final boolean silent) {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+ if (Objects.isNull(commitContext) || !commitContext.hasWriterProgress()) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ }
+ return false;
+ }
+ if (!isActive) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ }
+ return false;
+ }
+ return true;
+ }
+
+ public boolean ack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "ack", false)
+ && ackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean ackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ if (!acked.get()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for ack",
+ this,
+ commitContext);
+ }
+ return null;
+ }
+
+ if (ev.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: event {} already committed", this, commitContext);
+ ev.cleanUp(false);
+ return null;
+ }
+
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to advance commit frontier for {}",
+ this,
+ commitContext);
+ return ev;
+ }
+
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+
+ return acked.get();
+ }
+
+ public boolean nack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "nack", false)
+ && nackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ /**
+ * Silent version of ack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean ackSilent(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "ack", true)) {
+ return false;
+ }
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ return ev;
+ }
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+ return acked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private WriterId extractCommitWriterId(final SubscriptionCommitContext commitContext) {
+ final WriterId writerId = commitContext.getWriterId();
+ return Objects.nonNull(writerId) ? writerId : new WriterId(consensusGroupId.toString(), -1);
+ }
+
+ private WriterProgress extractCommitWriterProgress(
+ final SubscriptionCommitContext commitContext) {
+ return commitContext.getWriterProgress();
+ }
+
+ /**
+ * Silent version of nack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean nackSilent(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "nack", true)) {
+ return false;
+ }
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+ return nacked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean nackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for nack",
+ this,
+ commitContext);
+ return null;
+ }
+
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+
+ return nacked.get();
+ }
+
+ // ======================== Recycle ========================
+
+ /** Recycles in-flight events that are pollable (timed out) back to the prefetching queue. */
+ private void recycleInFlightEvents() {
+ for (final Pair<String, SubscriptionCommitContext> key :
+ new ArrayList<>(inFlightEvents.keySet())) {
+ inFlightEvents.compute(
+ key,
+ (k, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ if (ev.pollable()) {
+ ev.nack();
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected during recycle "
+ + "(nackCount={}), force-acking event {}",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: recycled timed-out event {} back to prefetching queue",
+ this,
+ ev);
+ return null;
+ }
+ return ev;
+ });
+ }
+ }
+
+ // ======================== Cleanup ========================
+
+ public void cleanUp() {
+ acquireWriteLock();
+ try {
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ pendingEntries.clear();
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ closeSubscriptionWALIterator();
+
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ // ======================== Seek ========================
+
+ /**
+ * Seeks to the earliest available WAL position. The actual position depends on WAL retention: if
+ * old files have been reclaimed, the earliest available position may be later than 0.
+ */
+ public void seekToBeginning() {
+ seekToResolvedPosition(0L, new RegionProgress(Collections.emptyMap()), "beginning");
+ }
+
+ /**
+ * Seeks to the current WAL write position. After this, only newly written data will be consumed.
+ */
+ public void seekToEnd() {
+ seekToResolvedPosition(
+ consensusReqReader.getCurrentSearchIndex(), computeTailRegionProgress(), "end");
+ }
+
+ public void seekToRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress not supported (no WAL directory)",
+ this);
+ seekToBeginning();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, false);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgress");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekToRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ public void seekAfterRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress not supported (no WAL directory)",
+ this);
+ seekToEnd();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgressAfter");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekAfterRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private synchronized void seekToResolvedPosition(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason) {
+ final PendingSeekRequest request;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ // Fence old commit contexts immediately. The grouped reset itself is applied later by the
+ // prefetch worker so WAL state and queue state still move under the queue's serial context.
+ final boolean previousPrefetchInitialized = prefetchInitialized;
+ final long previousSeekGeneration = seekGeneration.get();
+ final long targetSeekGeneration = seekGeneration.incrementAndGet();
+ request =
+ new PendingSeekRequest(
+ targetSearchIndex,
+ committedRegionProgress,
+ seekReason,
+ previousPrefetchInitialized,
+ previousSeekGeneration,
+ targetSeekGeneration);
+ pendingSeekRequest = request;
+ prefetchInitialized = true;
+ } finally {
+ releaseWriteLock();
+ }
+
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.isNull(subtask)) {
+ failPendingSeekBeforeScheduling(request);
+ request.awaitCompletion();
+ return;
+ }
+
+ subtask.requestWakeupNow();
+ request.awaitCompletion();
+ }
+
+ private boolean applyPendingSeekRequestIfNecessary() {
+ final PendingSeekRequest request = pendingSeekRequest;
+ if (Objects.isNull(request)) {
+ return false;
+ }
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return pendingSeekRequest != null;
+ }
+ pendingSeekRequest = null;
+ if (isClosed || closeRequested) {
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s is closing while applying seek", this)));
+ return true;
+ }
+ applySeekResetUnderWriteLock(request);
+ request.complete();
+ return true;
+ } catch (final RuntimeException e) {
+ request.fail(e);
+ throw e;
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ public void abortPendingSeekForRuntimeStop() {
+ final PendingSeekRequest requestToFail;
+
+ acquireWriteLock();
+ try {
+ requestToFail = pendingSeekRequest;
+ if (Objects.isNull(requestToFail)) {
+ return;
+ }
+ pendingSeekRequest = null;
+ prefetchInitialized = requestToFail.previousPrefetchInitialized;
+ if (seekGeneration.get() == requestToFail.targetSeekGeneration) {
+ seekGeneration.set(requestToFail.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: aborted pending seek({}) during runtime stop, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ requestToFail.seekReason,
+ true,
+ requestToFail.previousPrefetchInitialized,
+ requestToFail.targetSeekGeneration,
+ requestToFail.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ requestToFail.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s runtime stopped before seek(%s) was applied",
+ this, requestToFail.seekReason)));
+ }
+
+ private void failPendingSeekBeforeScheduling(final PendingSeekRequest request) {
+ final boolean closing;
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return;
+ }
+ closing = isClosed || closeRequested;
+ pendingSeekRequest = null;
+ prefetchInitialized = request.previousPrefetchInitialized;
+ if (seekGeneration.get() == request.targetSeekGeneration) {
+ seekGeneration.set(request.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: failed to schedule seek({}) because {}, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ request.seekReason,
+ closing ? "the queue is closing" : "prefetch runtime is unavailable",
+ true,
+ request.previousPrefetchInitialized,
+ request.targetSeekGeneration,
+ request.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ closing
+ ? "ConsensusPrefetchingQueue %s is closing before seek(%s) can be scheduled"
+ : "ConsensusPrefetchingQueue %s cannot schedule seek(%s) because prefetch runtime is unavailable",
+ this,
+ request.seekReason)));
+ }
+
+ private void applySeekResetUnderWriteLock(final PendingSeekRequest request) {
+ // 1. Clean up all queued and in-flight events
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ // 2. Discard stale pending entries from in-memory queue
+ pendingEntries.clear();
+
+ // 3. Reset per-writer release state and source-level dedup frontiers.
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ if (Objects.nonNull(request.committedRegionProgress)
+ && !request.committedRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(request.committedRegionProgress);
+ }
+
+ // 4. Reset WAL read position
+ nextExpectedSearchIndex.set(request.targetSearchIndex);
+ requestSubscriptionWalReset(request.targetSearchIndex, seekGeneration.get());
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = seekGeneration.get();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+
+ // 5. Reset commit state to the writer progress immediately before the first re-delivered
+ // entry so seek/rebind resumes from the intended frontier.
+ commitManager.resetState(
+ brokerId, topicName, consensusGroupId, request.committedRegionProgress);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seek({}) applied to searchIndex={}, writerCount={}, seekGeneration={}",
+ this,
+ request.seekReason,
+ request.targetSearchIndex,
+ Objects.nonNull(request.committedRegionProgress)
+ ? request.committedRegionProgress.getWriterPositions().size()
+ : 0,
+ seekGeneration.get());
+ }
+
+ private RegionProgress computeTailRegionProgress() {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final Map<WriterId, WriterProgress> tailProgressByWriter = new LinkedHashMap<>();
+ final File[] walFiles = WALFileUtils.listAllWALFiles(walNode.getLogDirectory());
+ if (Objects.isNull(walFiles) || walFiles.length == 0) {
+ mergeTailProgress(tailProgressByWriter, walNode.getCurrentWALMetaDataSnapshot());
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ WALFileUtils.ascSortByVersionId(walFiles);
+ final long liveVersionId = walNode.getCurrentWALFileVersion();
+ final WALMetaData liveSnapshot = walNode.getCurrentWALMetaDataSnapshot();
+ for (final File walFile : walFiles) {
+ final long versionId = WALFileUtils.parseVersionId(walFile.getName());
+ if (versionId == liveVersionId) {
+ mergeTailProgress(tailProgressByWriter, liveSnapshot);
+ continue;
+ }
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ mergeTailProgress(tailProgressByWriter, reader.getMetaData());
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to read WAL metadata from {} while computing seekToEnd frontier",
+ this,
+ walFile,
+ e);
+ }
+ }
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ private void mergeTailProgress(
+ final Map<WriterId, WriterProgress> tailProgressByWriter, final WALMetaData metadata) {
+ if (Objects.isNull(metadata)) {
+ return;
+ }
+ final List<Long> physicalTimes = metadata.getPhysicalTimes();
+ final List<Short> nodeIds = metadata.getNodeIds();
+ final List<Long> localSeqs = metadata.getLocalSeqs();
+ final int size = Math.min(Math.min(physicalTimes.size(), nodeIds.size()), localSeqs.size());
+ for (int i = 0; i < size; i++) {
+ final int writerNodeId = nodeIds.get(i);
+ final long physicalTime = physicalTimes.get(i);
+ final long localSeq = localSeqs.get(i);
+ if (writerNodeId < 0 || physicalTime < 0L || localSeq < 0L) {
+ continue;
+ }
+
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), writerNodeId);
+ final WriterProgress candidateProgress = new WriterProgress(physicalTime, localSeq);
+ final WriterProgress currentProgress = tailProgressByWriter.get(writerId);
+ if (Objects.isNull(currentProgress)
+ || compareWriterProgress(candidateProgress, currentProgress) > 0) {
+ tailProgressByWriter.put(writerId, candidateProgress);
+ }
+ }
+ }
+
+ /**
+ * Extracts the maximum timestamp from an InsertNode. For row nodes this is the single timestamp;
+ * for tablet nodes, {@code times} is sorted so the last element is the max. For composite nodes,
+ * iterates over children.
+ *
+ * @return the maximum timestamp, or {@code Long.MIN_VALUE} if extraction fails
+ */
+ private long extractMaxTime(final InsertNode insertNode) {
+ try {
+ if (insertNode instanceof InsertRowNode) {
+ return ((InsertRowNode) insertNode).getTime();
+ }
+ if (insertNode instanceof InsertTabletNode) {
+ final InsertTabletNode tabletNode = (InsertTabletNode) insertNode;
+ final int rowCount = tabletNode.getRowCount();
+ return rowCount > 0 ? tabletNode.getTimes()[rowCount - 1] : Long.MIN_VALUE;
+ }
+ if (insertNode instanceof InsertMultiTabletsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertTabletNode child :
+ ((InsertMultiTabletsNode) insertNode).getInsertTabletNodeList()) {
+ final int rowCount = child.getRowCount();
+ if (rowCount > 0) {
+ max = Math.max(max, child.getTimes()[rowCount - 1]);
+ }
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row : ((InsertRowsNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsOfOneDeviceNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row :
+ ((InsertRowsOfOneDeviceNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ // Fallback: use getMinTime() which at least gets a timestamp
+ return insertNode.getMinTime();
+ } catch (final Exception e) {
+ return Long.MIN_VALUE;
+ }
+ }
+
+ /**
+ * Checks whether it is time to inject a watermark event and does so if the configured interval
+ * has elapsed. Called from prefetch rounds after processing data and during idle scheduling.
+ */
+ private void maybeInjectWatermark() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return; // No data observed yet, nothing to report
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0) {
+ return; // Watermark disabled
+ }
+ final long now = System.currentTimeMillis();
+ if (now - lastWatermarkEmitTimeMs >= intervalMs) {
+ injectWatermark(maxObservedTimestamp);
+ lastWatermarkEmitTimeMs = now;
+ }
+ }
+
+ /**
+ * Injects a {@link SubscriptionPollResponseType#WATERMARK} event into the prefetching queue. The
+ * committed mapping is deliberately NOT recorded because watermark events are metadata, not user
+ * data.
+ *
+ * @param watermarkTimestamp the maximum data timestamp observed so far
+ */
+ private void injectWatermark(final long watermarkTimestamp) {
+ final int dataNodeId = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final SubscriptionCommitContext watermarkCtx = createNonCommittableSeekContext(dataNodeId);
+ final SubscriptionEvent watermarkEvent =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.WATERMARK.getType(),
+ new WatermarkPayload(watermarkTimestamp, dataNodeId),
+ watermarkCtx);
+ prefetchingQueue.add(watermarkEvent);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: injected WATERMARK, watermarkTimestamp={}",
+ this,
+ watermarkTimestamp);
+ }
+
+ /** Returns the maximum observed data timestamp for metrics. */
+ public long getMaxObservedTimestamp() {
+ return maxObservedTimestamp;
+ }
+
+ private void markAcceptedFromPending() {
+ pendingPathAcceptedEntries.incrementAndGet();
+ }
+
+ private void markAcceptedFromWal() {
+ walPathAcceptedEntries.incrementAndGet();
+ }
+
+ public void close() {
+ final PendingSeekRequest seekRequestToFail;
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> prefetchBinding;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ closeRequested = true;
+ seekRequestToFail = pendingSeekRequest;
+ pendingSeekRequest = null;
+ } finally {
+ releaseWriteLock();
+ }
+
+ prefetchBinding = detachPrefetchSubtask();
+
+ if (Objects.nonNull(seekRequestToFail)) {
+ seekRequestToFail.fail(
+ new IllegalStateException(
+ String.format("ConsensusPrefetchingQueue %s is closing before seek applies", this)));
+ }
+
+ if (Objects.nonNull(prefetchBinding.right)) {
+ prefetchBinding.right.cancelPendingExecution();
+ prefetchBinding.right.awaitIdle();
+ }
+
+ try {
+ acquireWriteLock();
+ try {
+ if (!isClosed
+ && pendingSeekRequest == null
+ && seekGeneration.get() == observedSeekGeneration) {
+ flushLingeringBatchOnCloseUnderWriteLock();
+ }
+ markClosed();
+ } finally {
+ releaseWriteLock();
+ }
+
+ // Deregister metrics after the queue is fully closed.
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance()
+ .deregister(getPrefetchingQueueId());
+
+ if (Objects.nonNull(prefetchBinding.left) && Objects.nonNull(prefetchBinding.right)) {
+ if (!prefetchBinding.left.isShutdown()) {
+ prefetchBinding.left.deregister(prefetchBinding.right.getTaskId());
+ } else {
+ prefetchBinding.right.close();
+ }
+ }
+
+ try {
+ // Deregister from IoTConsensusServerImpl (stop receiving in-memory data).
+ serverImpl.deregisterSubscriptionQueue(pendingEntries);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error during deregister", this, e);
+ } finally {
+ try {
+ cleanUp();
+ } finally {
+ // Persist progress before closing
+ commitManager.persistAll();
+ }
+ }
+ } finally {
+ closeRequested = false;
+ }
+ }
+
+ private void flushLingeringBatchOnCloseUnderWriteLock() {
+ if (lingerBatch.isEmpty()) {
+ return;
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: flushing {} lingering tablets during close",
+ this,
+ lingerBatch.tablets.size());
+ if (!flushBatch(lingerBatch, observedSeekGeneration)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to flush lingering batch during close, discarding it",
+ this);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ }
+ }
+
+ private SubscriptionEvent generateErrorResponse(final String errorMessage) {
+ return new SubscriptionEvent(
+ SubscriptionPollResponseType.ERROR.getType(),
+ new ErrorPayload(errorMessage, false),
+ createNonCommittableContext(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()));
+ }
+
+ private SubscriptionEvent generateOutdatedErrorResponse() {
+ return new SubscriptionEvent(
+ SubscriptionPollResponseType.ERROR.getType(),
+ ErrorPayload.OUTDATED_ERROR_PAYLOAD,
+ createNonCommittableContext(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()));
+ }
+
+ /**
+ * Shared subscription events still use {@link SubscriptionCommitContext#INVALID_COMMIT_ID} to
+ * mark metadata and error payloads as non-committable. Consensus correctness never treats this
+ * sentinel as a replay or commit frontier.
+ */
+ private SubscriptionCommitContext createNonCommittableContext(final int dataNodeId) {
+ return new SubscriptionCommitContext(
+ dataNodeId,
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ INVALID_COMMIT_ID);
+ }
+
+ private SubscriptionCommitContext createNonCommittableSeekContext(final int dataNodeId) {
+ return new SubscriptionCommitContext(
+ dataNodeId,
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ INVALID_COMMIT_ID,
+ seekGeneration.get(),
+ consensusGroupId.toString());
+ }
+
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ return PipeDataNodeAgent.runtime().getRebootTimes() > commitContext.getRebootTimes()
+ || seekGeneration.get() != commitContext.getSeekGeneration();
+ }
+
+ // ======================== Status ========================
+
+ public boolean isClosed() {
+ return isClosed;
+ }
+
+ public void markClosed() {
+ isClosed = true;
+ }
+
+ // ======================== Routing Runtime Version Control ========================
+
+ public long getWalGapSkippedEntries() {
+ return walGapSkippedEntries.get();
+ }
+
+ public long getEpochChangeCount() {
+ return runtimeVersionChangeCount.get();
+ }
+
+ // ======================== Leader Activation ========================
+
+ /**
+ * Activates or deactivates this queue. Only the preferred-writer (leader) node's queue should be
+ * active. Inactive queues skip prefetching and return null on poll.
+ */
+ public void setActive(final boolean active) {
+ this.isActive = active;
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: isActive set to {} (region={})",
+ this,
+ active,
+ consensusGroupId);
+ if (active) {
+ requestPrefetch();
+ }
+ }
+
+ public boolean isActive() {
+ return isActive;
+ }
+
+ public void setActiveWriterNodeIds(final Set<Integer> activeWriterNodeIds) {
+ this.runtimeActiveWriterNodeIds =
+ Collections.unmodifiableSet(
+ new LinkedHashSet<>(Objects.requireNonNull(activeWriterNodeIds)));
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: runtimeActiveWriterNodeIds={}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, orderMode={}, preferredWriterNodeId={})",
+ this,
+ this.runtimeActiveWriterNodeIds,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ orderMode,
+ preferredWriterNodeId);
+ requestPrefetch();
+ }
+
+ private void refreshEffectiveActiveWriterNodeIds() {
+ final LinkedHashSet<Integer> effectiveWriterNodeIds = new LinkedHashSet<>();
+ switch (orderMode) {
+ case TopicConstant.ORDER_MODE_MULTI_WRITER_VALUE:
+ effectiveWriterNodeIds.addAll(runtimeActiveWriterNodeIds);
+ if (effectiveWriterNodeIds.isEmpty() && preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ break;
+ case TopicConstant.ORDER_MODE_PER_WRITER_VALUE:
+ if (preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ break;
+ case TopicConstant.ORDER_MODE_LEADER_ONLY_VALUE:
+ default:
+ if (preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ if (previousPreferredWriterNodeId >= 0
+ && previousPreferredWriterNodeId != preferredWriterNodeId
+ && runtimeActiveWriterNodeIds.contains(previousPreferredWriterNodeId)) {
+ effectiveWriterNodeIds.add(previousPreferredWriterNodeId);
+ }
+ break;
+ }
+ this.activeWriterNodeIds = Collections.unmodifiableSet(effectiveWriterNodeIds);
+ }
+
+ public void setPreferredWriterNodeId(final int preferredWriterNodeId) {
+ if (this.preferredWriterNodeId != preferredWriterNodeId) {
+ previousPreferredWriterNodeId = this.preferredWriterNodeId;
+ } else {
+ previousPreferredWriterNodeId = -1;
+ }
+ this.preferredWriterNodeId = preferredWriterNodeId;
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: preferredWriterNodeId set to {}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, orderMode={})",
+ this,
+ this.preferredWriterNodeId,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ orderMode);
+ requestPrefetch();
+ }
+
+ public Set<Integer> getActiveWriterNodeIds() {
+ return activeWriterNodeIds;
+ }
+
+ public void setOrderMode(final String orderMode) {
+ final String normalizedOrderMode = TopicConfig.normalizeOrderMode(orderMode);
+ if (Objects.equals(this.orderMode, normalizedOrderMode)) {
+ return;
+ }
+ this.orderMode = normalizedOrderMode;
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: orderMode set to {}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, preferredWriterNodeId={}, runtimeActiveWriterNodeIds={})",
+ this,
+ this.orderMode,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ preferredWriterNodeId,
+ runtimeActiveWriterNodeIds);
+ requestPrefetch();
+ }
+
+ public String getOrderMode() {
+ return orderMode;
+ }
+
+ private boolean isLaneRuntimeActive(final WriterLaneId laneId) {
+ final Set<Integer> writerNodeIds = activeWriterNodeIds;
+ return writerNodeIds.isEmpty() || writerNodeIds.contains(laneId.writerNodeId);
+ }
+
+ public void applyRuntimeState(final ConsensusRegionRuntimeState runtimeState) {
+ Objects.requireNonNull(runtimeState, "runtimeState");
+ this.runtimeVersion = runtimeState.getRuntimeVersion();
+ runtimeVersionChangeCount.incrementAndGet();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: applied runtimeVersion {}",
+ this,
+ runtimeState.getRuntimeVersion());
+ setPreferredWriterNodeId(runtimeState.getPreferredWriterNodeId());
+ setActiveWriterNodeIds(runtimeState.getActiveWriterNodeIds());
+ // "active" decides whether this replica should serve subscription traffic on the current node.
+ // In multi-writer mode, activeWriterNodeIds may intentionally include follower replicas for
+ // ordering/watermark coordination, so it must not be reused as the local service-activation
+ // signal.
+ setActive(runtimeState.isActive());
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: applied runtimeState={}, preferredWriterNodeId={}",
+ this,
+ runtimeState,
+ runtimeState.getPreferredWriterNodeId());
+ if (runtimeState.isActive()) {
+ requestPrefetch();
+ }
+ }
+
+ public String getPrefetchingQueueId() {
+ return brokerId + "_" + topicName;
+ }
+
+ public long getSubscriptionUncommittedEventCount() {
+ return inFlightEvents.size();
+ }
+
+ /** Exposes the current seek generation for runtime tests and metrics. */
+ public long getCurrentSeekGeneration() {
+ return seekGeneration.get();
+ }
+
+ public int getPrefetchedEventCount() {
+ return prefetchingQueue.size();
+ }
+
+ public long getCurrentReadSearchIndex() {
+ return nextExpectedSearchIndex.get();
+ }
+
+ public long getPendingPathAcceptedEntries() {
+ return pendingPathAcceptedEntries.get();
+ }
+
+ public long getWalPathAcceptedEntries() {
+ return walPathAcceptedEntries.get();
+ }
+
+ public String getBrokerId() {
+ return brokerId;
+ }
+
+ public String getTopicName() {
+ return topicName;
+ }
+
+ public ConsensusGroupId getConsensusGroupId() {
+ return consensusGroupId;
+ }
+
+ /**
+ * Returns an approximate backlog for this queue.
+ *
+ * <p>The metric intentionally avoids collapsing per-writer committed progress into a single
+ * scalar local sequence. Instead it counts queued/in-flight work and adds one extra unit when the
+ * local WAL reader still has unread entries beyond its current replay cursor.
+ */
+ public long getLag() {
Review Comment:
Rewrote the lag comment
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = inFlightEvents.get(new Pair<>(consumerId, commitContext));
+ if (Objects.isNull(event)) {
+ if (isCommitContextOutdated(commitContext)) {
+ return generateOutdatedErrorResponse();
+ }
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
+ this, consumerId, commitContext));
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ // ======================== Prefetch Round Drive ========================
+
+ private static final long WAL_GAP_RETRY_SLEEP_MS = 10L;
+ private static final long WAL_GAP_WAIT_LOG_INTERVAL_MS = 5_000L;
+
+ private static final long PREFETCH_STATS_LOG_INTERVAL_MS = 5_000L;
+
+ public PrefetchRoundResult drivePrefetchOnce() {
+ if (applyPendingSeekRequestIfNecessary()) {
+ return closeRequested ? PrefetchRoundResult.dormant() : PrefetchRoundResult.rescheduleNow();
+ }
+
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !prefetchInitialized) {
+ return PrefetchRoundResult.dormant();
+ }
+
+ logPeriodicStatsIfNecessary();
+
+ final long currentSeekGeneration = seekGeneration.get();
+ if (currentSeekGeneration != observedSeekGeneration) {
+ resetRoundStateForSeek(currentSeekGeneration);
+ }
+
+ applyPendingSubscriptionWalReset(observedSeekGeneration);
+ recycleInFlightEvents();
+
+ if (!isActive || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return computeIdleRoundResult();
+ }
+
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+ final int batchMaxDelayMs = config.getSubscriptionConsensusBatchMaxDelayInMs();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+
+ final List<IndexedConsensusRequest> batch = drainPendingEntries(maxWalEntries);
+ if (!batch.isEmpty()) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: drained {} entries from pendingEntries, "
+ + "first searchIndex={}, last searchIndex={}, nextExpected={}, "
+ + "prefetchingQueueSize={}",
+ this,
+ batch.size(),
+ batch.get(0).getSearchIndex(),
+ batch.get(batch.size() - 1).getSearchIndex(),
+ nextExpectedSearchIndex.get(),
+ prefetchingQueue.size());
+
+ final boolean batchAccepted =
+ accumulateFromPending(
+ batch, lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes);
+ if (!batchAccepted) {
+ if (pendingWalGapRetryRequested) {
+ // Once a drained batch hits an unresolved WAL gap, the affected suffix falls back to
+ // the WAL path on later rounds instead of being requeued into the bounded pending path.
+ return PrefetchRoundResult.rescheduleAfter(WAL_GAP_RETRY_SLEEP_MS);
+ }
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ }
+
+ if (batch.isEmpty() && lingerBatch.isEmpty()) {
+ tryCatchUpFromWAL(observedSeekGeneration);
+ }
+
+ if (!drainBufferedRealtimeLanes(
+ lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes)) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerElapsedMs = System.currentTimeMillis() - lingerBatch.firstTabletTimeMs;
+ if (lingerElapsedMs >= batchMaxDelayMs) {
+ if (seekGeneration.get() != observedSeekGeneration) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: time-based flush, {} tablets lingered for {}ms "
+ + "(threshold={}ms)",
+ this,
+ lingerBatch.tablets.size(),
+ lingerElapsedMs,
+ batchMaxDelayMs);
+ flushBatch(lingerBatch, observedSeekGeneration);
+ }
+ }
+
+ maybeInjectWatermark();
+ return computeIdleRoundResult();
+ } catch (final Throwable fatal) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: prefetch round failed " + "(type={}, message={})",
+ this,
+ fatal.getClass().getName(),
+ fatal.getMessage(),
+ fatal);
+ if (fatal instanceof VirtualMachineError) {
+ markClosed();
+ return PrefetchRoundResult.dormant();
+ }
+ return PrefetchRoundResult.rescheduleAfter(100L);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private void logPeriodicStatsIfNecessary() {
+ final long nowMs = System.currentTimeMillis();
+ if (nowMs - lastStatsLogTimeMs < PREFETCH_STATS_LOG_INTERVAL_MS) {
+ return;
+ }
+
+ final long currentPendingAcceptedEntries = pendingPathAcceptedEntries.get();
+ final long currentWalAcceptedEntries = walPathAcceptedEntries.get();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: periodic stats, lag={}, pendingDelta={}, walDelta={}, "
+ + "pendingTotal={}, walTotal={}, pendingQueueSize={}, prefetchingQueueSize={}, "
+ + "inFlightEventsSize={}, realtimeLaneCount={}, walHasNext={}, isActive={}, subtaskScheduled={}",
+ this,
+ getLag(),
+ currentPendingAcceptedEntries - lastPendingAcceptedEntries,
+ currentWalAcceptedEntries - lastWalAcceptedEntries,
+ currentPendingAcceptedEntries,
+ currentWalAcceptedEntries,
+ pendingEntries.size(),
+ prefetchingQueue.size(),
+ inFlightEvents.size(),
+ realtimeEntriesByLane.size(),
+ hasReadableWalEntries(),
+ isActive,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ lastStatsLogTimeMs = nowMs;
+ lastPendingAcceptedEntries = currentPendingAcceptedEntries;
+ lastWalAcceptedEntries = currentWalAcceptedEntries;
+ }
+
+ private void resetRoundStateForSeek(final long newSeekGeneration) {
+ restorePendingSubscriptionWalCursor(newSeekGeneration);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = newSeekGeneration;
+ }
+
+ private List<IndexedConsensusRequest> drainPendingEntries(final int maxWalEntries) {
+ final List<IndexedConsensusRequest> batch = new ArrayList<>();
+ IndexedConsensusRequest next;
+ while (batch.size() < maxWalEntries && (next = pendingEntries.poll()) != null) {
+ batch.add(next);
+ }
+ return batch;
+ }
+
+ private PrefetchRoundResult computeIdleRoundResult() {
+ if (isClosed || !prefetchInitialized || !isActive) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (hasImmediatePrefetchableWork()) {
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ long delayMs = Long.MAX_VALUE;
+ if (hasHistoricalWalLag()) {
+ delayMs = Math.min(delayMs, WAL_GAP_RETRY_SLEEP_MS);
+ }
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerDelayMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusBatchMaxDelayInMs()
+ - (System.currentTimeMillis() - lingerBatch.firstTabletTimeMs);
+ delayMs = Math.min(delayMs, Math.max(1L, lingerDelayMs));
+ }
+
+ final long watermarkDelayMs = computeWatermarkDelayMs();
+ if (watermarkDelayMs > 0L) {
+ delayMs = Math.min(delayMs, watermarkDelayMs);
+ }
+
+ if (!inFlightEvents.isEmpty()) {
+ delayMs =
+ Math.min(
+ delayMs,
+ SubscriptionConfig.getInstance().getSubscriptionRecycleUncommittedEventIntervalMs());
+ }
+
+ return delayMs == Long.MAX_VALUE
+ ? PrefetchRoundResult.dormant()
+ : PrefetchRoundResult.rescheduleAfter(delayMs);
+ }
+
+ private long computeWatermarkDelayMs() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return -1L;
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0L) {
+ return -1L;
+ }
+ if (lastWatermarkEmitTimeMs == 0L) {
+ return 1L;
+ }
+ final long elapsedMs = System.currentTimeMillis() - lastWatermarkEmitTimeMs;
+ return elapsedMs >= intervalMs ? 1L : Math.max(1L, intervalMs - elapsedMs);
+ }
+
+ private boolean hasImmediatePrefetchableWork() {
+ return !pendingEntries.isEmpty() || !realtimeEntriesByLane.isEmpty() || hasReadableWalEntries();
+ }
+
+ private boolean hasHistoricalWalLag() {
+ return nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex();
+ }
+
+ /**
+ * Accumulates tablets from pending entries into the linger buffer. When pending replay outruns
+ * the local WAL reader, this method backfills the local-index gap from WAL before continuing.
+ *
+ * @return false if the batch became stale because seek generation changed while flushing
+ */
+ private static boolean hasLocalSearchIndex(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() >= 0;
+ }
+
+ private boolean isBeforeLocalCursor(final IndexedConsensusRequest request) {
+ return hasLocalSearchIndex(request) && request.getSearchIndex() < nextExpectedSearchIndex.get();
+ }
+
+ private void advanceLocalCursorIfPresent(final IndexedConsensusRequest request) {
+ if (hasLocalSearchIndex(request)) {
+ nextExpectedSearchIndex.set(request.getSearchIndex() + 1);
+ }
+ }
+
+ private boolean appendRealtimeRequest(
+ final IndexedConsensusRequest request,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean fromPending) {
+ final PreparedEntry preparedEntry = prepareEntry(request);
+ if (Objects.isNull(preparedEntry)) {
+ return true;
+ }
+ if (!appendPreparedEntryViaRealtimeLane(
+ batchState, preparedEntry, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+ if (fromPending) {
+ markAcceptedFromPending();
+ } else {
+ markAcceptedFromWal();
+ }
+ return true;
+ }
+
+ private boolean accumulateFromPending(
+ final List<IndexedConsensusRequest> batch,
+ final DeliveryBatchState lingerBatch,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+
+ int processedCount = 0;
+ int skippedCount = 0;
+
+ for (int index = 0; index < batch.size(); index++) {
+ final IndexedConsensusRequest request = batch.get(index);
+ final long searchIndex = request.getSearchIndex();
+
+ // Only local-indexed requests participate in the internal WAL read cursor.
+ final long expected = nextExpectedSearchIndex.get();
+ if (hasLocalSearchIndex(request) && searchIndex > expected) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: gap detected, expected={}, got={}. "
+ + "Filling {} entries from WAL.",
+ this,
+ expected,
+ searchIndex,
+ searchIndex - expected);
+ if (!fillGapFromWAL(
+ expected,
+ searchIndex,
+ lingerBatch,
+ expectedSeekGeneration,
+ maxTablets,
+ maxBatchBytes)) {
+ return false;
+ }
+ }
+
+ if (isBeforeLocalCursor(request)) {
+ skippedCount++;
+ continue;
+ }
+
+ if (shouldSkipForRecoveryProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ request, lingerBatch, expectedSeekGeneration, maxTablets, maxBatchBytes, true)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(request);
+ processedCount++;
+ advanceLocalCursorIfPresent(request);
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: accumulate complete, batchSize={}, processed={}, "
+ + "skipped={}, lingerTablets={}, nextExpected={}",
+ this,
+ batch.size(),
+ processedCount,
+ skippedCount,
+ lingerBatch.tablets.size(),
+ nextExpectedSearchIndex.get());
+
+ return true;
+ }
+
+ /**
+ * Fills a gap in the pending queue by reading entries from WAL so the internal local replay
+ * cursor stays contiguous even when pending delivery jumps ahead of the WAL iterator.
+ *
+ * <p>Temporary WAL visibility lag is treated as a normal back-pressure condition: once a drained
+ * pending batch encounters an unresolved local-index gap, the queue backs off and lets the
+ * affected suffix fall back to the WAL path on later rounds. This keeps replay contiguous without
+ * requeueing the drained batch back into the bounded pending queue.
+ *
+ * @return false if gap fill had to stop because the current batch became stale or the queue was
+ * interrupted/closed
+ */
+ private boolean fillGapFromWAL(
+ final long fromIndex,
+ final long toIndex,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ pendingWalGapRetryRequested = false;
+ resetSubscriptionWALPosition(fromIndex);
+ if (seekGeneration.get() != expectedSeekGeneration || isClosed) {
+ return false;
+ }
+ if (!pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, Integer.MAX_VALUE, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final long nextExpected = nextExpectedSearchIndex.get();
+ if (nextExpected >= toIndex) {
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ return true;
+ }
+
+ final long nowMs = System.currentTimeMillis();
+ if (walGapWaitStartTimeMs == 0L) {
+ walGapWaitStartTimeMs = nowMs;
+ }
+ if (lastWalGapWaitLogTimeMs == 0L
+ || nowMs - lastWalGapWaitLogTimeMs >= WAL_GAP_WAIT_LOG_INTERVAL_MS) {
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: waiting {}ms for WAL gap [{}, {}) to become visible, "
+ + "currentNextExpected={}, currentWalIndex={}, seekGeneration={}",
+ this,
+ nowMs - walGapWaitStartTimeMs,
+ nextExpected,
+ toIndex,
+ nextExpected,
+ consensusReqReader.getCurrentSearchIndex(),
+ expectedSeekGeneration);
+ lastWalGapWaitLogTimeMs = nowMs;
+ }
+ onWalGapRetryScheduled();
+ pendingWalGapRetryRequested = true;
+ return false;
+ }
+
+ /**
+ * Try catch-up from WAL when the pending queue was empty. This handles cold-start or scenarios
+ * where the subscription started after data was already written.
+ */
+ private void tryCatchUpFromWAL(final long expectedSeekGeneration) {
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+
+ final DeliveryBatchState batchState = new DeliveryBatchState();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ final boolean accepted =
+ pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, maxWalEntries, maxTablets, maxBatchBytes);
+ if (!accepted) {
+ return;
+ }
+
+ if (!batchState.isEmpty()) {
+ flushBatch(batchState, expectedSeekGeneration);
+ }
+ }
+
+ private boolean pumpFromSubscriptionWAL(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxWalEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return true;
+ }
+
+ subscriptionWALIterator.refresh();
+ ensureSubscriptionWalReadable();
+
+ int entriesRead = 0;
+ while (entriesRead < maxWalEntries
+ && subscriptionWALIterator.hasNext()
+ && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ try {
+ final IndexedConsensusRequest walEntry = subscriptionWALIterator.next();
+ entriesRead++;
+
+ if (isBeforeLocalCursor(walEntry)) {
+ continue;
+ }
+ if (shouldSkipForRecoveryProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ walEntry, batchState, expectedSeekGeneration, maxTablets, maxBatchBytes, false)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(walEntry);
+ advanceLocalCursorIfPresent(walEntry);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error reading subscription WAL", this, e);
+ break;
+ }
+ }
+
+ if (entriesRead > 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL read {} entries, nextExpectedSearchIndex={}",
+ this,
+ entriesRead,
+ nextExpectedSearchIndex.get());
+ }
+ return true;
+ }
+
+ private void ensureSubscriptionWalReadable() {
+ if (Objects.isNull(subscriptionWALIterator)
+ || subscriptionWALIterator.hasNext()
+ || !(consensusReqReader instanceof WALNode)) {
+ return;
+ }
+
+ final long currentWalIndex = consensusReqReader.getCurrentSearchIndex();
+ if (nextExpectedSearchIndex.get() > currentWalIndex) {
+ return;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL exhausted at {} while current WAL is {}. "
+ + "Rolling WAL file to expose current-file entries.",
+ this,
+ nextExpectedSearchIndex.get(),
+ currentWalIndex);
+ ((WALNode) consensusReqReader).rollWALFile();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ if (Objects.nonNull(subscriptionWALIterator)) {
+ subscriptionWALIterator.refresh();
+ }
+ }
+
+ private void resetSubscriptionWALPosition(final long startSearchIndex) {
+ closeSubscriptionWALIterator();
+ subscriptionWALIterator = createSubscriptionWALIterator(startSearchIndex);
+ }
+
+ protected ProgressWALIterator createSubscriptionWALIterator(final long startSearchIndex) {
+ if (consensusReqReader instanceof WALNode) {
+ return new ProgressWALIterator((WALNode) consensusReqReader, startSearchIndex);
+ }
+ return null;
+ }
+
+ protected void onWalGapRetryScheduled() {}
+
+ private boolean hasReadableWalEntries() {
+ return Objects.nonNull(subscriptionWALIterator) && subscriptionWALIterator.hasNext();
+ }
+
+ private void requestSubscriptionWalReset(
+ final long targetSearchIndex, final long seekGenerationValue) {
+ pendingSubscriptionWalResetSearchIndex = targetSearchIndex;
+ pendingSubscriptionWalResetGeneration = seekGenerationValue;
+ }
+
+ private void applyPendingSubscriptionWalReset(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ resetSubscriptionWALPosition(pendingSubscriptionWalResetSearchIndex);
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ }
+
+ private void restorePendingSubscriptionWalCursor(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ // A seek can land in the middle of a prefetch iteration. Restore the local cursor to the
+ // pending seek target before resuming under the new generation so stale in-flight work does
+ // not permanently advance the historical replay frontier.
+ nextExpectedSearchIndex.set(pendingSubscriptionWalResetSearchIndex);
+ }
+
+ private void closeSubscriptionWALIterator() {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return;
+ }
+ try {
+ subscriptionWALIterator.close();
+ } catch (final IOException e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error closing subscription WAL iterator", this, e);
+ } finally {
+ subscriptionWALIterator = null;
+ }
+ }
+
+ /**
+ * Deserializes the IConsensusRequest entries within an IndexedConsensusRequest to produce an
+ * InsertNode. WAL entries are typically stored as IoTConsensusRequest (serialized ByteBuffers),
+ * and a single logical write may be split across multiple fragments (SearchNode). This method
+ * handles both cases.
+ *
+ * <p>The deserialization follows the same pattern as {@code
+ * DataRegionStateMachine.grabPlanNode()}.
+ */
+ private InsertNode deserializeToInsertNode(final IndexedConsensusRequest indexedRequest) {
+ final List<SearchNode> searchNodes = new ArrayList<>();
+ PlanNode nonSearchNode = null;
+
+ for (final IConsensusRequest req : indexedRequest.getRequests()) {
+ PlanNode planNode;
+ try {
+ if (req instanceof IoTConsensusRequest) {
+ // WAL entries read from file are wrapped as IoTConsensusRequest (ByteBuffer)
+ planNode = WALEntry.deserializeForConsensus(req.serializeToByteBuffer());
+ } else if (req instanceof InsertNode) {
+ // In-memory entries (not yet flushed to WAL file) may already be PlanNode
+ planNode = (PlanNode) req;
+ } else {
+ // ByteBufferConsensusRequest or unknown
+ planNode = PlanNodeType.deserialize(req.serializeToByteBuffer());
+ }
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to deserialize IConsensusRequest "
+ + "(type={}) in searchIndex={}: {}",
+ this,
+ req.getClass().getSimpleName(),
+ indexedRequest.getSearchIndex(),
+ e.getMessage(),
+ e);
+ continue;
+ }
+
+ if (planNode instanceof SearchNode) {
+ final SearchNode searchNode = (SearchNode) planNode;
+ searchNode.setSearchIndex(indexedRequest.getSearchIndex());
+ if (indexedRequest.getSyncIndex() >= 0) {
+ searchNode.setSyncIndex(indexedRequest.getSyncIndex());
+ }
+ if (indexedRequest.getPhysicalTime() > 0) {
+ searchNode.setPhysicalTime(indexedRequest.getPhysicalTime());
+ }
+ if (indexedRequest.getNodeId() >= 0) {
+ searchNode.setNodeId(indexedRequest.getNodeId());
+ }
+ searchNodes.add(searchNode);
+ } else {
+ nonSearchNode = planNode;
+ }
+ }
+
+ // Merge split SearchNode fragments (same pattern as DataRegionStateMachine.grabPlanNode)
+ if (!searchNodes.isEmpty()) {
+ final PlanNode merged = searchNodes.get(0).merge(searchNodes);
+ if (merged instanceof InsertNode) {
+ final InsertNode mergedInsert = (InsertNode) merged;
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: deserialized merged InsertNode for searchIndex={}, "
+ + "type={}, deviceId={}, searchNodeCount={}",
+ this,
+ indexedRequest.getSearchIndex(),
+ mergedInsert.getType(),
+ ConsensusLogToTabletConverter.safeDeviceIdForLog(mergedInsert),
+ searchNodes.size());
+
+ return mergedInsert;
+ }
+ }
+
+ if (nonSearchNode != null) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: searchIndex={} contains non-InsertNode PlanNode: {}",
+ this,
+ indexedRequest.getSearchIndex(),
+ nonSearchNode.getClass().getSimpleName());
+ }
+
+ return null;
+ }
+
+ private PreparedEntry prepareEntry(final IndexedConsensusRequest indexedRequest) {
+ final InsertNode insertNode = deserializeToInsertNode(indexedRequest);
+ if (Objects.isNull(insertNode)) {
+ return null;
+ }
+
+ final long localSeq =
+ indexedRequest.getProgressLocalSeq() >= 0
+ ? indexedRequest.getProgressLocalSeq()
+ : indexedRequest.getSearchIndex();
+ final long searchIndex = indexedRequest.getSearchIndex();
+ final long physicalTime =
+ indexedRequest.getPhysicalTime() > 0
+ ? indexedRequest.getPhysicalTime()
+ : insertNode.getPhysicalTime();
+ final int writerNodeId =
+ indexedRequest.getNodeId() >= 0 ? indexedRequest.getNodeId() : insertNode.getNodeId();
+
+ trackWriterLane(writerNodeId);
+ final long maxTs = extractMaxTime(insertNode);
+ if (maxTs > maxObservedTimestamp) {
+ maxObservedTimestamp = maxTs;
+ }
+ final List<Tablet> tablets = converter.convert(insertNode);
+ if (tablets.isEmpty()) {
+ return null;
+ }
+
+ return new PreparedEntry(tablets, physicalTime, writerNodeId, localSeq, searchIndex);
+ }
+
+ private static long estimateTabletSize(final Tablet tablet) {
+ return PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet);
+ }
+
+ private void createAndEnqueueEvent(
+ final List<Tablet> tablets, final long startSearchIndex, final long endSearchIndex) {
+ createAndEnqueueEvent(
+ tablets, startSearchIndex, endSearchIndex, endSearchIndex, seekGeneration.get());
+ }
+
+ private boolean createAndEnqueueEvent(
+ final List<Tablet> tablets,
+ final long startSearchIndex,
+ final long endSearchIndex,
+ final long commitLocalSeq,
+ final long expectedSeekGeneration) {
+ if (tablets.isEmpty()) {
+ return true;
+ }
+
+ if (seekGeneration.get() != expectedSeekGeneration) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: skip stale event with searchIndex range [{}, {}], "
+ + "expectedSeekGeneration={}, currentSeekGeneration={}",
+ this,
+ startSearchIndex,
+ endSearchIndex,
+ expectedSeekGeneration,
+ seekGeneration.get());
+ return false;
+ }
+
+ final SubscriptionCommitContext commitContext = buildWriterCommitContext(commitLocalSeq);
+ final WriterId writerId = commitContext.getWriterId();
+ final WriterProgress writerProgress = commitContext.getWriterProgress();
+ commitManager.recordMapping(brokerId, topicName, consensusGroupId, writerId, writerProgress);
+
+ // nextOffset <= 0 means all tablets delivered in single batch
+ // -tablets.size() indicates total count
+ // Use Map<String, List<Tablet>> constructor with actual database name for table model;
+ final TabletsPayload payload =
+ new TabletsPayload(
+ Collections.singletonMap(converter.getDatabaseName(), tablets), -tablets.size());
+
+ final SubscriptionEvent event =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.TABLETS.getType(), payload, commitContext);
+
+ prefetchingQueue.add(event);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: ENQUEUED event with {} tablets, "
+ + "searchIndex range [{}, {}], prefetchQueueSize={}",
+ this,
+ tablets.size(),
+ startSearchIndex,
+ endSearchIndex,
+ prefetchingQueue.size());
+
+ // After enqueuing the data event, control metadata is handled separately from user data.
+ return true;
+ }
+
+ private SubscriptionCommitContext buildWriterCommitContext(final long localSeq) {
+ final int effectiveNodeId =
+ batchWriterNodeId >= 0
+ ? batchWriterNodeId
+ : IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), effectiveNodeId);
+ final WriterProgress writerProgress = new WriterProgress(batchPhysicalTime, localSeq);
+ return new SubscriptionCommitContext(
+ IoTDBDescriptor.getInstance().getConfig().getDataNodeId(),
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ seekGeneration.get(),
+ writerId,
+ writerProgress);
+ }
+
+ private void updateBatchWriterProgress(final long physicalTime, final int writerNodeId) {
+ if (physicalTime > 0) {
+ this.batchPhysicalTime = physicalTime;
+ }
+ if (writerNodeId >= 0) {
+ this.batchWriterNodeId = writerNodeId;
+ }
+ }
+
+ private void resetBatchWriterProgress() {
+ this.batchPhysicalTime = 0L;
+ this.batchWriterNodeId = -1;
+ }
+
+ private long estimateTabletsBytes(final List<Tablet> tablets) {
+ long estimatedBytes = 0L;
+ for (final Tablet tablet : tablets) {
+ estimatedBytes += estimateTabletSize(tablet);
+ }
+ return estimatedBytes;
+ }
+
+ private boolean appendPreparedEntryViaRealtimeLane(
+ final DeliveryBatchState batchState,
+ final PreparedEntry preparedEntry,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ bufferRealtimeEntry(preparedEntry);
+ return drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes);
+ }
+
+ private int getRealtimeBufferedEntryCount() {
+ int count = 0;
+ for (final NavigableMap<Long, PreparedEntry> laneEntries : realtimeEntriesByLane.values()) {
+ count += laneEntries.size();
+ }
+ return count;
+ }
+
+ private boolean drainBufferedRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ while (!realtimeEntriesByLane.isEmpty()) {
+ final int bufferedBefore = getRealtimeBufferedEntryCount();
+ if (!drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final int bufferedAfter = getRealtimeBufferedEntryCount();
+ if (bufferedAfter == 0 || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return true;
+ }
+
+ if (batchState.isEmpty()) {
+ return true;
+ }
+
+ if (!flushBatch(batchState, expectedSeekGeneration)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private boolean canAppendLaneEntry(
+ final DeliveryBatchState batchState,
+ final LaneBufferedEntry entry,
+ final long entryEstimatedBytes,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ final boolean wouldExceedEntryLimit =
+ maxEntries != Integer.MAX_VALUE && batchState.entryCount >= maxEntries;
+ final boolean wouldExceedTabletLimit =
+ !batchState.isEmpty() && batchState.tablets.size() + entry.getTablets().size() > maxTablets;
+ final boolean wouldExceedByteLimit =
+ !batchState.isEmpty() && batchState.estimatedBytes + entryEstimatedBytes > maxBatchBytes;
+ // Keep all consensus subscription modes on a single-writer commit/delivery shape so
+ // SubscriptionCommitContext and RegionProgress remain per-writer.
+ final boolean writerChanged =
+ !batchState.isEmpty() && batchState.writerNodeId != entry.getWriterNodeId();
+ return !(wouldExceedEntryLimit
+ || wouldExceedTabletLimit
+ || wouldExceedByteLimit
+ || writerChanged);
+ }
+
+ private boolean drainRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ return drainLaneEntries(
+ batchState,
+ this::buildRealtimeLaneFrontiers,
+ this::peekRealtimeEntry,
+ entry -> true,
+ (laneId, entry) -> removeRealtimeEntry(laneId, entry.localSeq),
+ Integer.MAX_VALUE,
+ maxTablets,
+ maxBatchBytes,
+ true);
+ }
+
+ private <T extends LaneBufferedEntry> boolean drainLaneEntries(
+ final DeliveryBatchState batchState,
+ final Supplier<PriorityQueue<LaneFrontier>> frontierSupplier,
+ final Function<WriterLaneId, T> headSupplier,
+ final Predicate<T> releasePredicate,
+ final BiConsumer<WriterLaneId, T> removeHeadAction,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean trackLingerTime) {
+ while (true) {
+ final PriorityQueue<LaneFrontier> frontiers = frontierSupplier.get();
+ if (frontiers.isEmpty()) {
+ return true;
+ }
+ final LaneFrontier frontier = frontiers.peek();
+ if (Objects.isNull(frontier) || frontier.isBarrier) {
+ return true;
+ }
+ final T laneHead = headSupplier.apply(frontier.laneId);
+ if (Objects.isNull(laneHead)) {
+ return true;
+ }
+ if (!releasePredicate.test(laneHead)) {
+ return true;
+ }
+
+ final long entryEstimatedBytes = estimateTabletsBytes(laneHead.getTablets());
+ if (!canAppendLaneEntry(
+ batchState, laneHead, entryEstimatedBytes, maxEntries, maxTablets, maxBatchBytes)) {
+ return true;
+ }
+
+ removeHeadAction.accept(frontier.laneId, laneHead);
+ batchState.append(laneHead, entryEstimatedBytes, trackLingerTime);
+ }
+ }
+
+ private boolean flushBatch(
+ final DeliveryBatchState batchState, final long expectedSeekGeneration) {
+ updateBatchWriterProgress(batchState.physicalTime, batchState.writerNodeId);
+ if (!createAndEnqueueEvent(
+ new ArrayList<>(batchState.tablets),
+ batchState.startSearchIndex,
+ batchState.endSearchIndex,
+ batchState.lastLocalSeq,
+ expectedSeekGeneration)) {
+ return false;
+ }
+ resetBatchWriterProgress();
+ batchState.reset();
+ return true;
+ }
+
+ // ======================== Commit (Ack/Nack) ========================
+
+ private boolean canAcceptCommitContext(
+ final SubscriptionCommitContext commitContext, final String action, final boolean silent) {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+ if (Objects.isNull(commitContext) || !commitContext.hasWriterProgress()) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ }
+ return false;
+ }
+ if (!isActive) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ }
+ return false;
+ }
+ return true;
+ }
+
+ public boolean ack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "ack", false)
+ && ackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean ackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ if (!acked.get()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for ack",
+ this,
+ commitContext);
+ }
+ return null;
+ }
+
+ if (ev.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: event {} already committed", this, commitContext);
+ ev.cleanUp(false);
+ return null;
+ }
+
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to advance commit frontier for {}",
+ this,
+ commitContext);
+ return ev;
+ }
+
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+
+ return acked.get();
+ }
+
+ public boolean nack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "nack", false)
+ && nackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ /**
+ * Silent version of ack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean ackSilent(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "ack", true)) {
+ return false;
+ }
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ return ev;
+ }
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+ return acked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private WriterId extractCommitWriterId(final SubscriptionCommitContext commitContext) {
+ final WriterId writerId = commitContext.getWriterId();
+ return Objects.nonNull(writerId) ? writerId : new WriterId(consensusGroupId.toString(), -1);
+ }
+
+ private WriterProgress extractCommitWriterProgress(
+ final SubscriptionCommitContext commitContext) {
+ return commitContext.getWriterProgress();
+ }
+
+ /**
+ * Silent version of nack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean nackSilent(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "nack", true)) {
+ return false;
+ }
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+ return nacked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean nackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for nack",
+ this,
+ commitContext);
+ return null;
+ }
+
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+
+ return nacked.get();
+ }
+
+ // ======================== Recycle ========================
+
+ /** Recycles in-flight events that are pollable (timed out) back to the prefetching queue. */
+ private void recycleInFlightEvents() {
+ for (final Pair<String, SubscriptionCommitContext> key :
+ new ArrayList<>(inFlightEvents.keySet())) {
+ inFlightEvents.compute(
+ key,
+ (k, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ if (ev.pollable()) {
+ ev.nack();
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected during recycle "
+ + "(nackCount={}), force-acking event {}",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: recycled timed-out event {} back to prefetching queue",
+ this,
+ ev);
+ return null;
+ }
+ return ev;
+ });
+ }
+ }
+
+ // ======================== Cleanup ========================
+
+ public void cleanUp() {
+ acquireWriteLock();
+ try {
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ pendingEntries.clear();
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ closeSubscriptionWALIterator();
+
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ // ======================== Seek ========================
+
+ /**
+ * Seeks to the earliest available WAL position. The actual position depends on WAL retention: if
+ * old files have been reclaimed, the earliest available position may be later than 0.
+ */
+ public void seekToBeginning() {
+ seekToResolvedPosition(0L, new RegionProgress(Collections.emptyMap()), "beginning");
+ }
+
+ /**
+ * Seeks to the current WAL write position. After this, only newly written data will be consumed.
+ */
+ public void seekToEnd() {
+ seekToResolvedPosition(
+ consensusReqReader.getCurrentSearchIndex(), computeTailRegionProgress(), "end");
+ }
+
+ public void seekToRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress not supported (no WAL directory)",
+ this);
+ seekToBeginning();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, false);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgress");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekToRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ public void seekAfterRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress not supported (no WAL directory)",
+ this);
+ seekToEnd();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgressAfter");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekAfterRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private synchronized void seekToResolvedPosition(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason) {
+ final PendingSeekRequest request;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ // Fence old commit contexts immediately. The grouped reset itself is applied later by the
+ // prefetch worker so WAL state and queue state still move under the queue's serial context.
+ final boolean previousPrefetchInitialized = prefetchInitialized;
+ final long previousSeekGeneration = seekGeneration.get();
+ final long targetSeekGeneration = seekGeneration.incrementAndGet();
+ request =
+ new PendingSeekRequest(
+ targetSearchIndex,
+ committedRegionProgress,
+ seekReason,
+ previousPrefetchInitialized,
+ previousSeekGeneration,
+ targetSeekGeneration);
+ pendingSeekRequest = request;
+ prefetchInitialized = true;
+ } finally {
+ releaseWriteLock();
+ }
+
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.isNull(subtask)) {
+ failPendingSeekBeforeScheduling(request);
+ request.awaitCompletion();
+ return;
+ }
+
+ subtask.requestWakeupNow();
+ request.awaitCompletion();
+ }
+
+ private boolean applyPendingSeekRequestIfNecessary() {
+ final PendingSeekRequest request = pendingSeekRequest;
+ if (Objects.isNull(request)) {
+ return false;
+ }
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return pendingSeekRequest != null;
+ }
+ pendingSeekRequest = null;
+ if (isClosed || closeRequested) {
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s is closing while applying seek", this)));
+ return true;
+ }
+ applySeekResetUnderWriteLock(request);
+ request.complete();
+ return true;
+ } catch (final RuntimeException e) {
+ request.fail(e);
+ throw e;
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ public void abortPendingSeekForRuntimeStop() {
+ final PendingSeekRequest requestToFail;
+
+ acquireWriteLock();
+ try {
+ requestToFail = pendingSeekRequest;
+ if (Objects.isNull(requestToFail)) {
+ return;
+ }
+ pendingSeekRequest = null;
+ prefetchInitialized = requestToFail.previousPrefetchInitialized;
+ if (seekGeneration.get() == requestToFail.targetSeekGeneration) {
+ seekGeneration.set(requestToFail.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: aborted pending seek({}) during runtime stop, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ requestToFail.seekReason,
+ true,
+ requestToFail.previousPrefetchInitialized,
+ requestToFail.targetSeekGeneration,
+ requestToFail.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ requestToFail.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s runtime stopped before seek(%s) was applied",
+ this, requestToFail.seekReason)));
+ }
+
+ private void failPendingSeekBeforeScheduling(final PendingSeekRequest request) {
+ final boolean closing;
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return;
+ }
+ closing = isClosed || closeRequested;
+ pendingSeekRequest = null;
+ prefetchInitialized = request.previousPrefetchInitialized;
+ if (seekGeneration.get() == request.targetSeekGeneration) {
+ seekGeneration.set(request.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: failed to schedule seek({}) because {}, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ request.seekReason,
+ closing ? "the queue is closing" : "prefetch runtime is unavailable",
+ true,
+ request.previousPrefetchInitialized,
+ request.targetSeekGeneration,
+ request.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ closing
+ ? "ConsensusPrefetchingQueue %s is closing before seek(%s) can be scheduled"
+ : "ConsensusPrefetchingQueue %s cannot schedule seek(%s) because prefetch runtime is unavailable",
+ this,
+ request.seekReason)));
+ }
+
+ private void applySeekResetUnderWriteLock(final PendingSeekRequest request) {
+ // 1. Clean up all queued and in-flight events
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ // 2. Discard stale pending entries from in-memory queue
+ pendingEntries.clear();
+
+ // 3. Reset per-writer release state and source-level dedup frontiers.
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ if (Objects.nonNull(request.committedRegionProgress)
+ && !request.committedRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(request.committedRegionProgress);
+ }
+
+ // 4. Reset WAL read position
+ nextExpectedSearchIndex.set(request.targetSearchIndex);
+ requestSubscriptionWalReset(request.targetSearchIndex, seekGeneration.get());
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = seekGeneration.get();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+
+ // 5. Reset commit state to the writer progress immediately before the first re-delivered
+ // entry so seek/rebind resumes from the intended frontier.
+ commitManager.resetState(
+ brokerId, topicName, consensusGroupId, request.committedRegionProgress);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seek({}) applied to searchIndex={}, writerCount={}, seekGeneration={}",
+ this,
+ request.seekReason,
+ request.targetSearchIndex,
+ Objects.nonNull(request.committedRegionProgress)
+ ? request.committedRegionProgress.getWriterPositions().size()
+ : 0,
+ seekGeneration.get());
+ }
+
+ private RegionProgress computeTailRegionProgress() {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final Map<WriterId, WriterProgress> tailProgressByWriter = new LinkedHashMap<>();
+ final File[] walFiles = WALFileUtils.listAllWALFiles(walNode.getLogDirectory());
+ if (Objects.isNull(walFiles) || walFiles.length == 0) {
+ mergeTailProgress(tailProgressByWriter, walNode.getCurrentWALMetaDataSnapshot());
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ WALFileUtils.ascSortByVersionId(walFiles);
+ final long liveVersionId = walNode.getCurrentWALFileVersion();
+ final WALMetaData liveSnapshot = walNode.getCurrentWALMetaDataSnapshot();
+ for (final File walFile : walFiles) {
+ final long versionId = WALFileUtils.parseVersionId(walFile.getName());
+ if (versionId == liveVersionId) {
+ mergeTailProgress(tailProgressByWriter, liveSnapshot);
+ continue;
+ }
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ mergeTailProgress(tailProgressByWriter, reader.getMetaData());
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to read WAL metadata from {} while computing seekToEnd frontier",
+ this,
+ walFile,
+ e);
+ }
+ }
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ private void mergeTailProgress(
+ final Map<WriterId, WriterProgress> tailProgressByWriter, final WALMetaData metadata) {
+ if (Objects.isNull(metadata)) {
+ return;
+ }
+ final List<Long> physicalTimes = metadata.getPhysicalTimes();
+ final List<Short> nodeIds = metadata.getNodeIds();
+ final List<Long> localSeqs = metadata.getLocalSeqs();
+ final int size = Math.min(Math.min(physicalTimes.size(), nodeIds.size()), localSeqs.size());
+ for (int i = 0; i < size; i++) {
+ final int writerNodeId = nodeIds.get(i);
+ final long physicalTime = physicalTimes.get(i);
+ final long localSeq = localSeqs.get(i);
+ if (writerNodeId < 0 || physicalTime < 0L || localSeq < 0L) {
+ continue;
+ }
+
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), writerNodeId);
+ final WriterProgress candidateProgress = new WriterProgress(physicalTime, localSeq);
+ final WriterProgress currentProgress = tailProgressByWriter.get(writerId);
+ if (Objects.isNull(currentProgress)
+ || compareWriterProgress(candidateProgress, currentProgress) > 0) {
+ tailProgressByWriter.put(writerId, candidateProgress);
+ }
+ }
+ }
+
+ /**
+ * Extracts the maximum timestamp from an InsertNode. For row nodes this is the single timestamp;
+ * for tablet nodes, {@code times} is sorted so the last element is the max. For composite nodes,
+ * iterates over children.
+ *
+ * @return the maximum timestamp, or {@code Long.MIN_VALUE} if extraction fails
+ */
+ private long extractMaxTime(final InsertNode insertNode) {
+ try {
+ if (insertNode instanceof InsertRowNode) {
+ return ((InsertRowNode) insertNode).getTime();
+ }
+ if (insertNode instanceof InsertTabletNode) {
+ final InsertTabletNode tabletNode = (InsertTabletNode) insertNode;
+ final int rowCount = tabletNode.getRowCount();
+ return rowCount > 0 ? tabletNode.getTimes()[rowCount - 1] : Long.MIN_VALUE;
+ }
+ if (insertNode instanceof InsertMultiTabletsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertTabletNode child :
+ ((InsertMultiTabletsNode) insertNode).getInsertTabletNodeList()) {
+ final int rowCount = child.getRowCount();
+ if (rowCount > 0) {
+ max = Math.max(max, child.getTimes()[rowCount - 1]);
+ }
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row : ((InsertRowsNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsOfOneDeviceNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row :
+ ((InsertRowsOfOneDeviceNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ // Fallback: use getMinTime() which at least gets a timestamp
+ return insertNode.getMinTime();
+ } catch (final Exception e) {
+ return Long.MIN_VALUE;
+ }
+ }
+
+ /**
+ * Checks whether it is time to inject a watermark event and does so if the configured interval
+ * has elapsed. Called from prefetch rounds after processing data and during idle scheduling.
+ */
+ private void maybeInjectWatermark() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return; // No data observed yet, nothing to report
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0) {
+ return; // Watermark disabled
+ }
+ final long now = System.currentTimeMillis();
+ if (now - lastWatermarkEmitTimeMs >= intervalMs) {
+ injectWatermark(maxObservedTimestamp);
+ lastWatermarkEmitTimeMs = now;
+ }
+ }
+
+ /**
+ * Injects a {@link SubscriptionPollResponseType#WATERMARK} event into the prefetching queue. The
+ * committed mapping is deliberately NOT recorded because watermark events are metadata, not user
+ * data.
+ *
+ * @param watermarkTimestamp the maximum data timestamp observed so far
+ */
+ private void injectWatermark(final long watermarkTimestamp) {
+ final int dataNodeId = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final SubscriptionCommitContext watermarkCtx = createNonCommittableSeekContext(dataNodeId);
+ final SubscriptionEvent watermarkEvent =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.WATERMARK.getType(),
+ new WatermarkPayload(watermarkTimestamp, dataNodeId),
+ watermarkCtx);
+ prefetchingQueue.add(watermarkEvent);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: injected WATERMARK, watermarkTimestamp={}",
+ this,
+ watermarkTimestamp);
+ }
+
+ /** Returns the maximum observed data timestamp for metrics. */
+ public long getMaxObservedTimestamp() {
+ return maxObservedTimestamp;
+ }
+
+ private void markAcceptedFromPending() {
+ pendingPathAcceptedEntries.incrementAndGet();
+ }
+
+ private void markAcceptedFromWal() {
+ walPathAcceptedEntries.incrementAndGet();
+ }
+
+ public void close() {
+ final PendingSeekRequest seekRequestToFail;
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> prefetchBinding;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ closeRequested = true;
+ seekRequestToFail = pendingSeekRequest;
+ pendingSeekRequest = null;
+ } finally {
+ releaseWriteLock();
+ }
+
+ prefetchBinding = detachPrefetchSubtask();
+
+ if (Objects.nonNull(seekRequestToFail)) {
+ seekRequestToFail.fail(
+ new IllegalStateException(
+ String.format("ConsensusPrefetchingQueue %s is closing before seek applies", this)));
+ }
+
+ if (Objects.nonNull(prefetchBinding.right)) {
+ prefetchBinding.right.cancelPendingExecution();
+ prefetchBinding.right.awaitIdle();
+ }
+
+ try {
+ acquireWriteLock();
+ try {
+ if (!isClosed
+ && pendingSeekRequest == null
+ && seekGeneration.get() == observedSeekGeneration) {
+ flushLingeringBatchOnCloseUnderWriteLock();
+ }
+ markClosed();
+ } finally {
+ releaseWriteLock();
+ }
+
+ // Deregister metrics after the queue is fully closed.
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance()
+ .deregister(getPrefetchingQueueId());
+
+ if (Objects.nonNull(prefetchBinding.left) && Objects.nonNull(prefetchBinding.right)) {
+ if (!prefetchBinding.left.isShutdown()) {
+ prefetchBinding.left.deregister(prefetchBinding.right.getTaskId());
+ } else {
+ prefetchBinding.right.close();
+ }
+ }
+
+ try {
+ // Deregister from IoTConsensusServerImpl (stop receiving in-memory data).
+ serverImpl.deregisterSubscriptionQueue(pendingEntries);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error during deregister", this, e);
+ } finally {
+ try {
+ cleanUp();
+ } finally {
+ // Persist progress before closing
+ commitManager.persistAll();
+ }
+ }
+ } finally {
+ closeRequested = false;
+ }
+ }
+
+ private void flushLingeringBatchOnCloseUnderWriteLock() {
+ if (lingerBatch.isEmpty()) {
+ return;
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: flushing {} lingering tablets during close",
+ this,
+ lingerBatch.tablets.size());
+ if (!flushBatch(lingerBatch, observedSeekGeneration)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to flush lingering batch during close, discarding it",
+ this);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ }
+ }
+
+ private SubscriptionEvent generateErrorResponse(final String errorMessage) {
+ return new SubscriptionEvent(
+ SubscriptionPollResponseType.ERROR.getType(),
+ new ErrorPayload(errorMessage, false),
+ createNonCommittableContext(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()));
+ }
+
+ private SubscriptionEvent generateOutdatedErrorResponse() {
+ return new SubscriptionEvent(
+ SubscriptionPollResponseType.ERROR.getType(),
+ ErrorPayload.OUTDATED_ERROR_PAYLOAD,
+ createNonCommittableContext(IoTDBDescriptor.getInstance().getConfig().getDataNodeId()));
+ }
+
+ /**
+ * Shared subscription events still use {@link SubscriptionCommitContext#INVALID_COMMIT_ID} to
+ * mark metadata and error payloads as non-committable. Consensus correctness never treats this
+ * sentinel as a replay or commit frontier.
+ */
+ private SubscriptionCommitContext createNonCommittableContext(final int dataNodeId) {
+ return new SubscriptionCommitContext(
+ dataNodeId,
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ INVALID_COMMIT_ID);
+ }
+
+ private SubscriptionCommitContext createNonCommittableSeekContext(final int dataNodeId) {
+ return new SubscriptionCommitContext(
+ dataNodeId,
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ INVALID_COMMIT_ID,
+ seekGeneration.get(),
+ consensusGroupId.toString());
+ }
+
+ public boolean isCommitContextOutdated(final SubscriptionCommitContext commitContext) {
+ return PipeDataNodeAgent.runtime().getRebootTimes() > commitContext.getRebootTimes()
+ || seekGeneration.get() != commitContext.getSeekGeneration();
+ }
+
+ // ======================== Status ========================
+
+ public boolean isClosed() {
+ return isClosed;
+ }
+
+ public void markClosed() {
+ isClosed = true;
+ }
+
+ // ======================== Routing Runtime Version Control ========================
+
+ public long getWalGapSkippedEntries() {
+ return walGapSkippedEntries.get();
+ }
+
+ public long getEpochChangeCount() {
+ return runtimeVersionChangeCount.get();
+ }
+
+ // ======================== Leader Activation ========================
+
+ /**
+ * Activates or deactivates this queue. Only the preferred-writer (leader) node's queue should be
+ * active. Inactive queues skip prefetching and return null on poll.
+ */
+ public void setActive(final boolean active) {
+ this.isActive = active;
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: isActive set to {} (region={})",
+ this,
+ active,
+ consensusGroupId);
+ if (active) {
+ requestPrefetch();
+ }
+ }
+
+ public boolean isActive() {
+ return isActive;
+ }
+
+ public void setActiveWriterNodeIds(final Set<Integer> activeWriterNodeIds) {
+ this.runtimeActiveWriterNodeIds =
+ Collections.unmodifiableSet(
+ new LinkedHashSet<>(Objects.requireNonNull(activeWriterNodeIds)));
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: runtimeActiveWriterNodeIds={}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, orderMode={}, preferredWriterNodeId={})",
+ this,
+ this.runtimeActiveWriterNodeIds,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ orderMode,
+ preferredWriterNodeId);
+ requestPrefetch();
+ }
+
+ private void refreshEffectiveActiveWriterNodeIds() {
+ final LinkedHashSet<Integer> effectiveWriterNodeIds = new LinkedHashSet<>();
+ switch (orderMode) {
+ case TopicConstant.ORDER_MODE_MULTI_WRITER_VALUE:
+ effectiveWriterNodeIds.addAll(runtimeActiveWriterNodeIds);
+ if (effectiveWriterNodeIds.isEmpty() && preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ break;
+ case TopicConstant.ORDER_MODE_PER_WRITER_VALUE:
+ if (preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ break;
+ case TopicConstant.ORDER_MODE_LEADER_ONLY_VALUE:
+ default:
+ if (preferredWriterNodeId >= 0) {
+ effectiveWriterNodeIds.add(preferredWriterNodeId);
+ }
+ if (previousPreferredWriterNodeId >= 0
+ && previousPreferredWriterNodeId != preferredWriterNodeId
+ && runtimeActiveWriterNodeIds.contains(previousPreferredWriterNodeId)) {
+ effectiveWriterNodeIds.add(previousPreferredWriterNodeId);
+ }
+ break;
+ }
+ this.activeWriterNodeIds = Collections.unmodifiableSet(effectiveWriterNodeIds);
+ }
+
+ public void setPreferredWriterNodeId(final int preferredWriterNodeId) {
+ if (this.preferredWriterNodeId != preferredWriterNodeId) {
+ previousPreferredWriterNodeId = this.preferredWriterNodeId;
+ } else {
+ previousPreferredWriterNodeId = -1;
+ }
+ this.preferredWriterNodeId = preferredWriterNodeId;
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: preferredWriterNodeId set to {}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, orderMode={})",
+ this,
+ this.preferredWriterNodeId,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ orderMode);
+ requestPrefetch();
+ }
+
+ public Set<Integer> getActiveWriterNodeIds() {
+ return activeWriterNodeIds;
+ }
+
+ public void setOrderMode(final String orderMode) {
+ final String normalizedOrderMode = TopicConfig.normalizeOrderMode(orderMode);
+ if (Objects.equals(this.orderMode, normalizedOrderMode)) {
+ return;
+ }
+ this.orderMode = normalizedOrderMode;
+ refreshEffectiveActiveWriterNodeIds();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: orderMode set to {}, effectiveActiveWriterNodeIds={} "
+ + "(region={}, preferredWriterNodeId={}, runtimeActiveWriterNodeIds={})",
+ this,
+ this.orderMode,
+ this.activeWriterNodeIds,
+ consensusGroupId,
+ preferredWriterNodeId,
+ runtimeActiveWriterNodeIds);
+ requestPrefetch();
+ }
+
+ public String getOrderMode() {
+ return orderMode;
+ }
+
+ private boolean isLaneRuntimeActive(final WriterLaneId laneId) {
+ final Set<Integer> writerNodeIds = activeWriterNodeIds;
+ return writerNodeIds.isEmpty() || writerNodeIds.contains(laneId.writerNodeId);
+ }
+
+ public void applyRuntimeState(final ConsensusRegionRuntimeState runtimeState) {
+ Objects.requireNonNull(runtimeState, "runtimeState");
+ this.runtimeVersion = runtimeState.getRuntimeVersion();
+ runtimeVersionChangeCount.incrementAndGet();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: applied runtimeVersion {}",
+ this,
+ runtimeState.getRuntimeVersion());
+ setPreferredWriterNodeId(runtimeState.getPreferredWriterNodeId());
+ setActiveWriterNodeIds(runtimeState.getActiveWriterNodeIds());
+ // "active" decides whether this replica should serve subscription traffic on the current node.
+ // In multi-writer mode, activeWriterNodeIds may intentionally include follower replicas for
+ // ordering/watermark coordination, so it must not be reused as the local service-activation
+ // signal.
+ setActive(runtimeState.isActive());
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: applied runtimeState={}, preferredWriterNodeId={}",
+ this,
+ runtimeState,
+ runtimeState.getPreferredWriterNodeId());
+ if (runtimeState.isActive()) {
+ requestPrefetch();
+ }
+ }
+
+ public String getPrefetchingQueueId() {
+ return brokerId + "_" + topicName;
+ }
+
+ public long getSubscriptionUncommittedEventCount() {
+ return inFlightEvents.size();
+ }
+
+ /** Exposes the current seek generation for runtime tests and metrics. */
+ public long getCurrentSeekGeneration() {
+ return seekGeneration.get();
+ }
+
+ public int getPrefetchedEventCount() {
+ return prefetchingQueue.size();
+ }
+
+ public long getCurrentReadSearchIndex() {
+ return nextExpectedSearchIndex.get();
+ }
+
+ public long getPendingPathAcceptedEntries() {
+ return pendingPathAcceptedEntries.get();
+ }
+
+ public long getWalPathAcceptedEntries() {
+ return walPathAcceptedEntries.get();
+ }
+
+ public String getBrokerId() {
+ return brokerId;
+ }
+
+ public String getTopicName() {
+ return topicName;
+ }
+
+ public ConsensusGroupId getConsensusGroupId() {
+ return consensusGroupId;
+ }
+
+ /**
+ * Returns an approximate backlog for this queue.
+ *
+ * <p>The metric intentionally avoids collapsing per-writer committed progress into a single
+ * scalar local sequence. Instead it counts queued/in-flight work and adds one extra unit when the
+ * local WAL reader still has unread entries beyond its current replay cursor.
+ */
+ public long getLag() {
+ long lag =
+ prefetchingQueue.size()
+ + inFlightEvents.size()
+ + pendingEntries.size()
+ + getRealtimeBufferedEntryCount();
+ if (nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex()) {
+ lag++;
+ }
Review Comment:
Made the unread-WAL part explicit with hasUnreadWalEntries instead of an unexplained lag++.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279202202
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/SubscriptionConsensusProgress.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Persisted commit metadata for a single (consumerGroup, topic, region) combination.
+ *
+ * <p>This object stores the committed per-writer region frontier plus the persistence throttling
+ * counter.
+ */
+public class SubscriptionConsensusProgress {
+
+ private volatile RegionProgress committedRegionProgress;
+
+ private final AtomicLong commitIndex;
Review Comment:
Renamed commitIndex to persistenceThrottleCounter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279147795
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279162731
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
Review Comment:
Currently not supported for consensus subscription.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "Caideyipi (via GitHub)" <gi...@apache.org>.
Caideyipi commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r2923211250
##########
iotdb-core/consensus/src/main/java/org/apache/iotdb/consensus/iot/IoTConsensusServerImpl.java:
##########
@@ -757,6 +805,41 @@ public long getSearchIndex() {
return searchIndex.get();
}
+ public ConsensusReqReader getConsensusReqReader() {
+ return consensusReqReader;
+ }
+
+ /**
+ * Registers a subscription pending queue for real-time in-memory data delivery. When {@link
+ * #write(IConsensusRequest)} succeeds, the IndexedConsensusRequest is offered to all registered
+ * subscription queues, enabling subscription consumers to receive data without waiting for WAL
+ * flush.
+ *
+ * @param queue the blocking queue to receive IndexedConsensusRequest entries
+ */
+ public void registerSubscriptionQueue(final BlockingQueue<IndexedConsensusRequest> queue) {
+ subscriptionQueues.add(queue);
+ // Immediately re-evaluate the safe delete index with new subscription awareness
+ checkAndUpdateSafeDeletedSearchIndex();
+ logger.info(
+ "Registered subscription queue for group {}, "
+ + "total subscription queues: {}, currentSearchIndex={}, this={}",
+ consensusGroupId,
+ subscriptionQueues.size(),
+ searchIndex.get(),
+ System.identityHashCode(this));
+ }
+
+ public void unregisterSubscriptionQueue(final BlockingQueue<IndexedConsensusRequest> queue) {
Review Comment:
deregister
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -200,41 +356,119 @@ public void bindPrefetchingQueue(final SubscriptionSinkSubtask subtask) {
prefetchingQueueCount.invalidate();
}
- public void updateCompletedTopicNames(final String consumerGroupId, final String topicName) {
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
+ public void bindConsensusPrefetchingQueue(
+ final String consumerGroupId,
+ final String topicName,
+ final String consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final long startSearchIndex) {
+ consumerGroupIdToConsensusBroker
+ .compute(
+ consumerGroupId,
+ (id, broker) -> {
+ if (Objects.isNull(broker)) {
+ LOGGER.info(
+ "Subscription: consensus broker bound to consumer group [{}] does not exist, create new for binding consensus prefetching queue",
+ consumerGroupId);
+ return new ConsensusSubscriptionBroker(consumerGroupId);
+ }
+ return broker;
+ })
+ .bindConsensusPrefetchingQueue(
+ topicName, consensusGroupId, serverImpl, converter, commitManager, startSearchIndex);
+ prefetchingQueueCount.invalidate();
+ }
+
+ public void unbindConsensusPrefetchingQueue(
+ final String consumerGroupId, final String topicName) {
+ final ConsensusSubscriptionBroker broker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
if (Objects.isNull(broker)) {
LOGGER.warn(
- "Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
+ "Subscription: consensus broker bound to consumer group [{}] does not exist",
+ consumerGroupId);
+ return;
+ }
+ broker.unbindConsensusPrefetchingQueue(topicName);
+ prefetchingQueueCount.invalidate();
+ }
+
+ public void unbindByRegion(final String regionId) {
+ int totalClosed = 0;
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ totalClosed += broker.unbindByRegion(regionId);
+ }
+ if (totalClosed > 0) {
+ prefetchingQueueCount.invalidate();
+ LOGGER.info(
+ "Subscription: unbound {} consensus prefetching queue(s) for removed region [{}]",
+ totalClosed,
+ regionId);
+ }
+ }
+
+ public void updateCompletedTopicNames(final String consumerGroupId, final String topicName) {
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
+ LOGGER.warn(
+ "Subscription: pipe broker bound to consumer group [{}] does not exist", consumerGroupId);
return;
}
- broker.updateCompletedTopicNames(topicName);
+ pipeBroker.updateCompletedTopicNames(topicName);
}
public void unbindPrefetchingQueue(final String consumerGroupId, final String topicName) {
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ // Try consensus broker first
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ consensusBroker.removeQueue(topicName);
Review Comment:
unbind？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3246260988
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -109,89 +201,319 @@ public List<SubscriptionCommitContext> commit(
final List<SubscriptionCommitContext> commitContexts,
final boolean nack) {
final String consumerGroupId = consumerConfig.getConsumerGroupId();
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ final String consumerId = consumerConfig.getConsumerId();
+ final List<SubscriptionCommitContext> allSuccessful = new ArrayList<>();
+
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+
+ if (Objects.isNull(pipeBroker) && Objects.isNull(consensusBroker)) {
+ final String errorMessage =
+ String.format("Subscription: no broker bound to consumer group [%s]", consumerGroupId);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionException(errorMessage);
+ }
+
+ // Partition commit contexts by which broker owns the topic.
+ final List<SubscriptionCommitContext> pipeContexts = new ArrayList<>();
+ final List<SubscriptionCommitContext> consensusContexts = new ArrayList<>();
+ for (final SubscriptionCommitContext ctx : commitContexts) {
+ final String topicName = ctx.getTopicName();
+ if (Objects.nonNull(consensusBroker)
+ && ConsensusSubscriptionSetupHandler.isConsensusBasedTopic(topicName)) {
+ consensusContexts.add(ctx);
+ } else {
+ pipeContexts.add(ctx);
+ }
+ }
+
+ if (Objects.nonNull(pipeBroker) && !pipeContexts.isEmpty()) {
+ allSuccessful.addAll(pipeBroker.commit(consumerId, pipeContexts, nack));
+ }
+ if (Objects.nonNull(consensusBroker) && !consensusContexts.isEmpty()) {
+ allSuccessful.addAll(consensusBroker.commit(consumerId, consensusContexts, nack));
+ }
+
+ return allSuccessful;
+ }
+
+ public void seek(
+ final ConsumerConfig consumerConfig, final String topicName, final short seekType) {
+ final String consumerGroupId = consumerConfig.getConsumerGroupId();
+
Review Comment:
Fixed. For consensus topics, non-owning DataNodes now treat seek/seekAfter as no-op success when no local queue exists, while owning DataNodes still apply the seek.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279204678
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/SubscriptionConsensusProgress.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Persisted commit metadata for a single (consumerGroup, topic, region) combination.
+ *
+ * <p>This object stores the committed per-writer region frontier plus the persistence throttling
+ * counter.
+ */
+public class SubscriptionConsensusProgress {
+
+ private volatile RegionProgress committedRegionProgress;
+
+ private final AtomicLong commitIndex;
+
+ public SubscriptionConsensusProgress() {
+ this(new RegionProgress(Collections.emptyMap()), 0L);
+ }
+
+ public SubscriptionConsensusProgress(
+ final RegionProgress committedRegionProgress, final long commitIndex) {
+ this.committedRegionProgress = normalize(committedRegionProgress);
+ this.commitIndex = new AtomicLong(commitIndex);
+ }
+
+ public RegionProgress getCommittedRegionProgress() {
+ return committedRegionProgress;
+ }
+
+ public void setCommittedRegionProgress(final RegionProgress committedRegionProgress) {
+ this.committedRegionProgress = normalize(committedRegionProgress);
+ }
+
+ public WriterId getCommittedWriterId() {
+ return getDerivedCommittedWriterState().writerId;
+ }
+
+ public WriterProgress getCommittedWriterProgress() {
+ return getDerivedCommittedWriterState().writerProgress;
+ }
+
+ public long getCommitIndex() {
+ return commitIndex.get();
+ }
+
+ public void incrementCommitIndex() {
+ commitIndex.incrementAndGet();
+ }
+
+ public void serialize(final DataOutputStream stream) throws IOException {
+ committedRegionProgress.serialize(stream);
+ ReadWriteIOUtils.write(commitIndex.get(), stream);
+ }
+
+ public static SubscriptionConsensusProgress deserialize(final ByteBuffer buffer) {
+ final RegionProgress committedRegionProgress = RegionProgress.deserialize(buffer);
+ final long commitIndex = ReadWriteIOUtils.readLong(buffer);
+ return new SubscriptionConsensusProgress(committedRegionProgress, commitIndex);
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ final SubscriptionConsensusProgress that = (SubscriptionConsensusProgress) o;
+ return commitIndex.get() == that.commitIndex.get()
+ && Objects.equals(committedRegionProgress, that.committedRegionProgress);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(committedRegionProgress, commitIndex.get());
+ }
+
+ @Override
+ public String toString() {
+ return "SubscriptionConsensusProgress{"
+ + "committedRegionProgress="
+ + committedRegionProgress
+ + ", commitIndex="
+ + commitIndex.get()
+ + '}';
+ }
+
+ private static RegionProgress normalize(final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+ final Map<WriterId, WriterProgress> normalized = new LinkedHashMap<>();
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ committedRegionProgress.getWriterPositions().entrySet()) {
+ if (Objects.nonNull(entry.getKey()) && Objects.nonNull(entry.getValue())) {
+ normalized.put(entry.getKey(), entry.getValue());
+ }
+ }
+ return new RegionProgress(normalized);
+ }
+
+ private DerivedCommittedWriterState getDerivedCommittedWriterState() {
+ WriterId bestWriterId = null;
+ WriterProgress bestWriterProgress = null;
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ committedRegionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(bestWriterProgress)
+ || compareWriterPosition(
+ entry.getKey(), entry.getValue(), bestWriterId, bestWriterProgress)
+ > 0) {
+ bestWriterId = entry.getKey();
+ bestWriterProgress = entry.getValue();
+ }
+ }
+ return new DerivedCommittedWriterState(
+ bestWriterId,
Review Comment:
Renamed bestWriterId and bestWriterProgress to latestWriterId and latestWriterProgress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279133462
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/agent/SubscriptionBrokerAgent.java:
##########
@@ -200,41 +522,183 @@ public void bindPrefetchingQueue(final SubscriptionSinkSubtask subtask) {
prefetchingQueueCount.invalidate();
}
- public void updateCompletedTopicNames(final String consumerGroupId, final String topicName) {
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
+ public void bindConsensusPrefetchingQueue(
+ final String consumerGroupId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ consumerGroupIdToConsensusBroker
+ .compute(
+ consumerGroupId,
+ (id, broker) -> {
+ if (Objects.isNull(broker)) {
+ LOGGER.info(
+ "Subscription: consensus broker bound to consumer group [{}] does not exist, create new for binding consensus prefetching queue",
+ consumerGroupId);
+ return new ConsensusSubscriptionBroker(consumerGroupId);
+ }
+ return broker;
+ })
+ .bindConsensusPrefetchingQueue(
+ topicName,
+ orderMode,
+ consensusGroupId,
+ serverImpl,
+ retentionPolicy,
+ converter,
+ commitManager,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+ prefetchingQueueCount.invalidate();
+ }
+
+ public void refreshConsensusQueueOrderMode(final String topicName, final String orderMode) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: refreshing consensus queue order-mode for topic [{}] to [{}]",
+ topicName,
+ orderMode);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.refreshConsensusQueueOrderMode(topicName, orderMode);
+ }
+ }
+
+ public void unbindConsensusPrefetchingQueue(
+ final String consumerGroupId, final String topicName) {
+ final ConsensusSubscriptionBroker broker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
if (Objects.isNull(broker)) {
LOGGER.warn(
- "Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
+ "Subscription: consensus broker bound to consumer group [{}] does not exist",
+ consumerGroupId);
return;
}
- broker.updateCompletedTopicNames(topicName);
+ broker.unbindConsensusPrefetchingQueue(topicName);
+ prefetchingQueueCount.invalidate();
+ }
+
+ public void unbindByRegion(final ConsensusGroupId regionId) {
+ int totalClosed = 0;
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ totalClosed += broker.unbindByRegion(regionId);
+ }
+ if (totalClosed > 0) {
+ prefetchingQueueCount.invalidate();
+ LOGGER.info(
+ "Subscription: unbound {} consensus prefetching queue(s) for removed region [{}]",
+ totalClosed,
+ regionId);
+ }
+ }
+
+ /**
+ * Activates or deactivates all consensus prefetching queues bound to {@code regionId} across all
+ * consumer groups. Called on leader migration to ensure only the preferred writer serves
+ * subscription data.
+ */
+ public void setActiveForRegion(final ConsensusGroupId regionId, final boolean active) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: setActiveForRegion regionId={}, active={}", regionId, active);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.setActiveForRegion(regionId, active);
+ }
+ }
+
+ public void setActiveWritersForRegion(
+ final ConsensusGroupId regionId, final Set<Integer> activeWriterNodeIds) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: setActiveWritersForRegion regionId={}, activeWriterNodeIds={}",
+ regionId,
+ activeWriterNodeIds);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.setActiveWritersForRegion(regionId, activeWriterNodeIds);
+ }
+ }
+
+ public void applyRuntimeStateForRegion(
+ final ConsensusGroupId regionId, final ConsensusRegionRuntimeState runtimeState) {
+ LOGGER.info(
+ "SubscriptionBrokerAgent: applyRuntimeStateForRegion regionId={}, runtimeState={}",
+ regionId,
+ runtimeState);
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.applyRuntimeStateForRegion(regionId, runtimeState);
+ }
+ }
+
+ public void abortConsensusPendingSeeksForRuntimeStop() {
+ for (final ConsensusSubscriptionBroker broker : consumerGroupIdToConsensusBroker.values()) {
+ broker.abortPendingSeeksForRuntimeStop();
+ }
+ }
+
+ public void updateCompletedTopicNames(final String consumerGroupId, final String topicName) {
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
+ LOGGER.warn(
+ "Subscription: pipe broker bound to consumer group [{}] does not exist", consumerGroupId);
+ return;
+ }
+ pipeBroker.updateCompletedTopicNames(topicName);
}
public void unbindPrefetchingQueue(final String consumerGroupId, final String topicName) {
- final SubscriptionBroker broker = consumerGroupIdToSubscriptionBroker.get(consumerGroupId);
- if (Objects.isNull(broker)) {
+ // Try consensus broker first
+ final ConsensusSubscriptionBroker consensusBroker =
+ consumerGroupIdToConsensusBroker.get(consumerGroupId);
+ if (Objects.nonNull(consensusBroker) && consensusBroker.hasQueue(topicName)) {
+ consensusBroker.unbindConsensusPrefetchingQueue(topicName);
+ prefetchingQueueCount.invalidate();
+ return;
+ }
+ // Fall back to pipe broker
+ final SubscriptionBroker pipeBroker = consumerGroupIdToPipeBroker.get(consumerGroupId);
+ if (Objects.isNull(pipeBroker)) {
LOGGER.warn(
"Subscription: broker bound to consumer group [{}] does not exist", consumerGroupId);
return;
}
- broker.unbindPrefetchingQueue(topicName);
+ pipeBroker.unbindPrefetchingQueue(topicName);
prefetchingQueueCount.invalidate();
Review Comment:
Reduced the duplicated pipe/consensus branches by selecting an ISubscriptionBroker for shared operations. I kept mode-specific lifecycle paths separate because one consumer group may own both a pipe broker and a consensus broker, so a single broker map would require a larger composite-broker refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279171992
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusPrefetchingQueue.java:
##########
@@ -0,0 +1,3443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.consensus.ConsensusGroupId;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.request.IConsensusRequest;
+import org.apache.iotdb.commons.subscription.config.SubscriptionConfig;
+import org.apache.iotdb.consensus.common.request.IndexedConsensusRequest;
+import org.apache.iotdb.consensus.common.request.IoTConsensusRequest;
+import org.apache.iotdb.consensus.iot.IoTConsensusServerImpl;
+import org.apache.iotdb.consensus.iot.SubscriptionWalRetentionPolicy;
+import org.apache.iotdb.consensus.iot.WriterSafeFrontierTracker;
+import org.apache.iotdb.consensus.iot.log.ConsensusReqReader;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.pipe.agent.PipeDataNodeAgent;
+import org.apache.iotdb.db.pipe.resource.memory.PipeMemoryWeightUtil;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.SearchNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.buffer.WALEntry;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.ProgressWALReader;
+import org.apache.iotdb.db.storageengine.dataregion.wal.io.WALMetaData;
+import org.apache.iotdb.db.storageengine.dataregion.wal.node.WALNode;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALFileUtils;
+import org.apache.iotdb.db.subscription.event.SubscriptionEvent;
+import org.apache.iotdb.db.subscription.metric.ConsensusSubscriptionPrefetchingQueueMetrics;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutor;
+import org.apache.iotdb.db.subscription.task.execution.ConsensusSubscriptionPrefetchExecutorManager;
+import org.apache.iotdb.db.subscription.task.subtask.ConsensusPrefetchSubtask;
+import org.apache.iotdb.rpc.subscription.config.TopicConfig;
+import org.apache.iotdb.rpc.subscription.config.TopicConstant;
+import org.apache.iotdb.rpc.subscription.payload.poll.ErrorPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.RegionProgress;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext;
+import org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionPollResponseType;
+import org.apache.iotdb.rpc.subscription.payload.poll.TabletsPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WatermarkPayload;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterId;
+import org.apache.iotdb.rpc.subscription.payload.poll.WriterProgress;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.write.record.Tablet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.LinkedBlockingDeque;
+import java.util.concurrent.PriorityBlockingQueue;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.BiConsumer;
+import java.util.function.Function;
+import java.util.function.Predicate;
+import java.util.function.Supplier;
+
+import static org.apache.iotdb.rpc.subscription.payload.poll.SubscriptionCommitContext.INVALID_COMMIT_ID;
+
+public class ConsensusPrefetchingQueue {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusPrefetchingQueue.class);
+
+ private final String brokerId; // consumer group id
+ private final String topicName;
+ private final ConsensusGroupId consensusGroupId;
+
+ private final IoTConsensusServerImpl serverImpl;
+
+ private final ConsensusReqReader consensusReqReader;
+
+ private final SubscriptionWalRetentionPolicy retentionPolicy;
+
+ private final WakeableIndexedConsensusQueue pendingEntries;
+
+ private static final int PENDING_QUEUE_CAPACITY = 4096;
+
+ private final ConsensusLogToTabletConverter converter;
+
+ private final ConsensusSubscriptionCommitManager commitManager;
+
+ private final AtomicLong seekGeneration;
+
+ /** Internal WAL reader cursor used only for local replay positioning and deduplication. */
+ private final AtomicLong nextExpectedSearchIndex;
+
+ private final PriorityBlockingQueue<SubscriptionEvent> prefetchingQueue;
+
+ private final Map<Pair<String, SubscriptionCommitContext>, SubscriptionEvent> inFlightEvents;
+
+ private static final int MAX_PREFETCHING_QUEUE_SIZE =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusPrefetchingQueueCapacity();
+
+ private final AtomicLong walGapSkippedEntries = new AtomicLong(0);
+
+ /** Guards queue state transitions that touch replay positioning, seek state, and lane buffers. */
+ private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
+
+ private volatile boolean isClosed = false;
+
+ private volatile boolean closeRequested = false;
+
+ private volatile boolean isActive = true;
+
+ private volatile Set<Integer> activeWriterNodeIds = Collections.emptySet();
+
+ private volatile Set<Integer> runtimeActiveWriterNodeIds = Collections.emptySet();
+
+ private volatile int preferredWriterNodeId = -1;
+
+ private volatile int previousPreferredWriterNodeId = -1;
+
+ // ======================== Routing Runtime Version ========================
+
+ private volatile long runtimeVersion = 0;
+
+ private final AtomicLong runtimeVersionChangeCount = new AtomicLong(0);
+
+ // ======================== Unified WAL / Release State ========================
+
+ private volatile ProgressWALIterator subscriptionWALIterator;
+
+ /**
+ * Seek requests must not close/reset the WAL iterator from RPC threads because the prefetch
+ * worker may be reading it concurrently. Instead, seek only records the latest desired reset and
+ * the queue's next prefetch round applies it after observing the new seek generation.
+ */
+ private volatile long pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+
+ private volatile long pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+
+ // ======================== Watermark ========================
+
+ /** Maximum data timestamp observed across all InsertNodes processed by this queue. */
+ private volatile long maxObservedTimestamp = Long.MIN_VALUE;
+
+ /** Wall-clock time (ms) of last watermark injection. 0 means never injected. */
+ private volatile long lastWatermarkEmitTimeMs = 0;
+
+ /** Number of entries accepted from realtime pending queue. */
+ private final AtomicLong pendingPathAcceptedEntries = new AtomicLong(0);
+
+ /** Number of entries accepted from WAL-backed paths (historical or catch-up). */
+ private final AtomicLong walPathAcceptedEntries = new AtomicLong(0);
+
+ private final Object prefetchBindingLock = new Object();
+
+ private volatile ConsensusPrefetchSubtask prefetchSubtask;
+
+ private volatile ConsensusSubscriptionPrefetchExecutor prefetchExecutor;
+
+ /**
+ * Whether the prefetch runtime has been initialized. Starts as false (dormant). Set to true on
+ * the first poll with a region progress hint or when a seek installs a pending reset. This keeps
+ * queue creation cheap: realtime entries can be buffered immediately while WAL replay state is
+ * only built once the queue is actually activated.
+ */
+ private volatile boolean prefetchInitialized = false;
+
+ private volatile PendingSeekRequest pendingSeekRequest;
+
+ private final DeliveryBatchState lingerBatch = new DeliveryBatchState();
+
+ private volatile long observedSeekGeneration;
+
+ private volatile long lastStatsLogTimeMs = System.currentTimeMillis();
+
+ private volatile long lastPendingAcceptedEntries = 0L;
+
+ private volatile long lastWalAcceptedEntries = 0L;
+
+ private volatile boolean pendingWalGapRetryRequested = false;
+
+ private volatile long walGapWaitStartTimeMs = 0L;
+
+ private volatile long lastWalGapWaitLogTimeMs = 0L;
+
+ /** Fallback committed region progress from local persisted state. */
+ private final RegionProgress fallbackCommittedRegionProgress;
+
+ /** Recovery-time per-writer frontiers used to skip already committed entries after restart. */
+ private final Map<WriterId, WriterProgress> recoveryWriterProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Source-level dedup frontier for follower-origin entries that do not carry a local searchIndex.
+ * The same request may first arrive through pendingEntries and later become visible from WAL;
+ * once a follower-origin localSeq has already been materialized into queue state, the WAL path
+ * must not materialize it again.
+ */
+ private final Map<WriterLaneId, Long> materializedFollowerProgressByWriter =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Lane state keyed by writer identity. Release gating reasons in terms of writer lanes and safe
+ * frontiers instead of a region-level committed frontier.
+ */
+ private final Map<WriterLaneId, WriterLaneState> writerLanes = new ConcurrentHashMap<>();
+
+ /**
+ * Realtime lane buffers used by both pending replay and WAL catch-up so queue materialization
+ * converges on the same per-writer lane representation before batch delivery.
+ */
+ private final Map<WriterLaneId, NavigableMap<Long, PreparedEntry>> realtimeEntriesByLane =
+ new ConcurrentHashMap<>();
+
+ /**
+ * Local tail position used only when initialization starts without any persisted region progress.
+ */
+ private final long fallbackTailSearchIndex;
+
+ /** Local sequence used to represent the position immediately before a writer's first record. */
+ private static final long BEFORE_FIRST_LOCAL_SEQ = -1L;
+
+ /** Writer-progress metadata for the current pending/WAL batch being assembled. */
+ private volatile long batchPhysicalTime = 0L;
+
+ private volatile int batchWriterNodeId = -1;
+ private volatile String orderMode = TopicConstant.ORDER_MODE_DEFAULT_VALUE;
+
+ protected enum ReplayLocateStatus {
+ FOUND,
+ AT_END,
+ LOCATE_MISS
+ }
+
+ protected static final class ReplayLocateDecision {
+ private final ReplayLocateStatus status;
+ private final long startSearchIndex;
+ private final RegionProgress recoveryRegionProgress;
+ private final String detail;
+
+ private ReplayLocateDecision(
+ final ReplayLocateStatus status,
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ this.status = status;
+ this.startSearchIndex = startSearchIndex;
+ this.recoveryRegionProgress = recoveryRegionProgress;
+ this.detail = detail;
+ }
+
+ static ReplayLocateDecision found(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.FOUND, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision atEnd(
+ final long startSearchIndex,
+ final RegionProgress recoveryRegionProgress,
+ final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.AT_END, startSearchIndex, recoveryRegionProgress, detail);
+ }
+
+ static ReplayLocateDecision locateMiss(
+ final RegionProgress recoveryRegionProgress, final String detail) {
+ return new ReplayLocateDecision(
+ ReplayLocateStatus.LOCATE_MISS, Long.MIN_VALUE, recoveryRegionProgress, detail);
+ }
+
+ protected ReplayLocateStatus getStatus() {
+ return status;
+ }
+
+ protected long getStartSearchIndex() {
+ return startSearchIndex;
+ }
+
+ protected RegionProgress getRecoveryRegionProgress() {
+ return recoveryRegionProgress;
+ }
+
+ protected String getDetail() {
+ return detail;
+ }
+ }
+
+ private static final class WakeableIndexedConsensusQueue
+ extends LinkedBlockingDeque<IndexedConsensusRequest> {
+
+ private final Runnable wakeupHook;
+
+ private WakeableIndexedConsensusQueue(final int capacity, final Runnable wakeupHook) {
+ super(capacity);
+ this.wakeupHook = wakeupHook;
+ }
+
+ @Override
+ public boolean offer(final IndexedConsensusRequest request) {
+ final boolean offered = super.offer(request);
+ if (offered) {
+ wakeupHook.run();
+ }
+ return offered;
+ }
+
+ @Override
+ public void put(final IndexedConsensusRequest request) throws InterruptedException {
+ super.put(request);
+ wakeupHook.run();
+ }
+ }
+
+ private static final class PendingSeekRequest {
+
+ private final long targetSearchIndex;
+ private final RegionProgress committedRegionProgress;
+ private final String seekReason;
+ private final boolean previousPrefetchInitialized;
+ private final long previousSeekGeneration;
+ private final long targetSeekGeneration;
+
+ private boolean completed = false;
+ private RuntimeException failure;
+
+ private PendingSeekRequest(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason,
+ final boolean previousPrefetchInitialized,
+ final long previousSeekGeneration,
+ final long targetSeekGeneration) {
+ this.targetSearchIndex = targetSearchIndex;
+ this.committedRegionProgress = committedRegionProgress;
+ this.seekReason = seekReason;
+ this.previousPrefetchInitialized = previousPrefetchInitialized;
+ this.previousSeekGeneration = previousSeekGeneration;
+ this.targetSeekGeneration = targetSeekGeneration;
+ }
+
+ private synchronized void complete() {
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void fail(final RuntimeException failure) {
+ this.failure = failure;
+ completed = true;
+ notifyAll();
+ }
+
+ private synchronized void awaitCompletion() {
+ while (!completed) {
+ try {
+ wait(50L);
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException("Interrupted while waiting for seek application", e);
+ }
+ }
+ if (failure != null) {
+ throw failure;
+ }
+ }
+ }
+
+ public ConsensusPrefetchingQueue(
+ final String brokerId,
+ final String topicName,
+ final String orderMode,
+ final ConsensusGroupId consensusGroupId,
+ final IoTConsensusServerImpl serverImpl,
+ final SubscriptionWalRetentionPolicy retentionPolicy,
+ final ConsensusLogToTabletConverter converter,
+ final ConsensusSubscriptionCommitManager commitManager,
+ final RegionProgress fallbackCommittedRegionProgress,
+ final long tailStartSearchIndex,
+ final long initialRuntimeVersion,
+ final boolean initialActive) {
+ this.brokerId = brokerId;
+ this.topicName = topicName;
+ this.consensusGroupId = consensusGroupId;
+ this.serverImpl = serverImpl;
+ this.consensusReqReader = serverImpl.getConsensusReqReader();
+ this.retentionPolicy = retentionPolicy;
+ this.converter = converter;
+ this.commitManager = commitManager;
+ this.fallbackCommittedRegionProgress = fallbackCommittedRegionProgress;
+ this.fallbackTailSearchIndex = tailStartSearchIndex;
+ this.runtimeVersion = initialRuntimeVersion;
+ this.isActive = initialActive;
+ this.orderMode = TopicConfig.normalizeOrderMode(orderMode);
+
+ this.seekGeneration = new AtomicLong(0);
+ this.nextExpectedSearchIndex = new AtomicLong(tailStartSearchIndex);
+
+ this.prefetchingQueue = new PriorityBlockingQueue<>();
+ this.inFlightEvents = new ConcurrentHashMap<>();
+ this.observedSeekGeneration = seekGeneration.get();
+
+ // Register pending queue early so we don't miss real-time writes
+ this.pendingEntries =
+ new WakeableIndexedConsensusQueue(PENDING_QUEUE_CAPACITY, this::requestPrefetch);
+ serverImpl.registerSubscriptionQueue(pendingEntries, retentionPolicy);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue created (dormant): brokerId={}, topicName={}, "
+ + "orderMode={}, consensusGroupId={}, fallbackCommittedRegionProgress={}, "
+ + "fallbackTailSearchIndex={}, initialRuntimeVersion={}, initialActive={}",
+ brokerId,
+ topicName,
+ this.orderMode,
+ consensusGroupId,
+ fallbackCommittedRegionProgress,
+ tailStartSearchIndex,
+ initialRuntimeVersion,
+ initialActive);
+
+ // Register metrics
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance().register(this);
+ }
+
+ // ======================== Lock Operations ========================
+
+ private void acquireReadLock() {
+ lock.readLock().lock();
+ }
+
+ private void releaseReadLock() {
+ lock.readLock().unlock();
+ }
+
+ private void acquireWriteLock() {
+ lock.writeLock().lock();
+ }
+
+ private void releaseWriteLock() {
+ lock.writeLock().unlock();
+ }
+
+ private void requestPrefetch() {
+ if (closeRequested || isClosed) {
+ return;
+ }
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.nonNull(subtask)) {
+ subtask.requestWakeupNow();
+ }
+ }
+
+ private ConsensusPrefetchSubtask ensurePrefetchSubtaskBound() {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return null;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ if (Objects.nonNull(currentSubtask)
+ && prefetchExecutor == currentExecutor
+ && !currentSubtask.isClosed()) {
+ return currentSubtask;
+ }
+
+ synchronized (prefetchBindingLock) {
+ if (closeRequested || isClosed) {
+ return null;
+ }
+
+ if (Objects.nonNull(prefetchSubtask)
+ && prefetchExecutor == currentExecutor
+ && !prefetchSubtask.isClosed()) {
+ return prefetchSubtask;
+ }
+
+ final ConsensusPrefetchSubtask staleSubtask = prefetchSubtask;
+ final ConsensusSubscriptionPrefetchExecutor staleExecutor = prefetchExecutor;
+ if (Objects.nonNull(staleSubtask)
+ && Objects.nonNull(staleExecutor)
+ && (staleExecutor != currentExecutor || staleSubtask.isClosed())
+ && !staleExecutor.isShutdown()) {
+ staleExecutor.deregister(staleSubtask.getTaskId());
+ }
+
+ final ConsensusPrefetchSubtask newSubtask = new ConsensusPrefetchSubtask(this);
+ if (!currentExecutor.register(newSubtask)) {
+ return null;
+ }
+ prefetchExecutor = currentExecutor;
+ prefetchSubtask = newSubtask;
+ return newSubtask;
+ }
+ }
+
+ private Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask>
+ detachPrefetchSubtask() {
+ synchronized (prefetchBindingLock) {
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> detached =
+ new Pair<>(prefetchExecutor, prefetchSubtask);
+ prefetchExecutor = null;
+ prefetchSubtask = null;
+ return detached;
+ }
+ }
+
+ private boolean shouldRecoverPrefetchBindingAfterEmptyPoll() {
+ if (!prefetchInitialized || isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+
+ final ConsensusSubscriptionPrefetchExecutor currentExecutor =
+ ConsensusSubscriptionPrefetchExecutorManager.getInstance().getExecutor();
+ if (Objects.isNull(currentExecutor)) {
+ return false;
+ }
+
+ final ConsensusPrefetchSubtask currentSubtask = prefetchSubtask;
+ final boolean bindingMissing =
+ Objects.isNull(currentSubtask)
+ || currentSubtask.isClosed()
+ || Objects.isNull(prefetchExecutor)
+ || prefetchExecutor.isShutdown()
+ || prefetchExecutor != currentExecutor;
+ if (!bindingMissing) {
+ return false;
+ }
+
+ return hasImmediatePrefetchableWork()
+ || hasHistoricalWalLag()
+ || !lingerBatch.isEmpty()
+ || !inFlightEvents.isEmpty()
+ || computeWatermarkDelayMs() > 0L;
+ }
+
+ // ======================== Poll ========================
+
+ public SubscriptionEvent poll(final String consumerId) {
+ return poll(consumerId, null);
+ }
+
+ public SubscriptionEvent poll(final String consumerId, final RegionProgress regionProgress) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !isActive) {
+ return null;
+ }
+ if (!prefetchInitialized) {
+ initPrefetch(regionProgress);
+ }
+ if (pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = pollInternal(consumerId);
+ if (Objects.nonNull(event) && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ requestPrefetch();
+ } else if (Objects.isNull(event) && shouldRecoverPrefetchBindingAfterEmptyPoll()) {
+ requestPrefetch();
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private synchronized void initPrefetch(final RegionProgress regionProgress) {
+ if (prefetchInitialized) {
+ return; // double-check under synchronization
+ }
+
+ final RegionProgress committedRegionProgress = resolveCommittedRegionProgressForInit();
+ final boolean useConsumerHint =
+ shouldUseConsumerRegionProgressHint(regionProgress, committedRegionProgress);
+ final RegionProgress recoveryRegionProgress =
+ useConsumerHint
+ ? mergeRecoveryRegionProgress(committedRegionProgress, regionProgress)
+ : committedRegionProgress;
+ final String progressSource =
+ useConsumerHint
+ ? Objects.nonNull(committedRegionProgress)
+ && !committedRegionProgress.getWriterPositions().isEmpty()
+ ? "merged committed region progress with consumer topic progress hint"
+ : "consumer topic progress hint"
+ : "committed region progress fallback";
+ final ReplayLocateDecision resolvedStart =
+ resolveInitReplayStartDecision(recoveryRegionProgress, progressSource);
+
+ clearRecoveryWriterProgress();
+ final RegionProgress effectiveRecoveryRegionProgress =
+ resolvedStart.getRecoveryRegionProgress();
+ if (Objects.nonNull(effectiveRecoveryRegionProgress)
+ && !effectiveRecoveryRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(effectiveRecoveryRegionProgress);
+ }
+
+ this.nextExpectedSearchIndex.set(resolvedStart.getStartSearchIndex());
+ if (consensusReqReader instanceof WALNode) {
+ this.subscriptionWALIterator =
+ new ProgressWALIterator(
+ (WALNode) consensusReqReader, resolvedStart.getStartSearchIndex());
+ }
+ this.prefetchInitialized = true;
+ this.observedSeekGeneration = seekGeneration.get();
+ this.lingerBatch.reset();
+ resetBatchWriterProgress();
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: prefetch initialized, startSearchIndex={}, progressSource={}, recoveryWriterCount={}",
+ this,
+ resolvedStart.getStartSearchIndex(),
+ resolvedStart.getDetail(),
+ recoveryWriterProgressByWriter.size());
+
+ requestPrefetch();
+ }
+
+ private ReplayLocateDecision resolveInitReplayStartDecision(
+ final RegionProgress recoveryRegionProgress, final String progressSource) {
+ if (Objects.isNull(recoveryRegionProgress)
+ || recoveryRegionProgress.getWriterPositions().isEmpty()) {
+ return ReplayLocateDecision.found(
+ fallbackTailSearchIndex,
+ new RegionProgress(Collections.emptyMap()),
+ progressSource + " (tail start without progress)");
+ }
+ if (!(consensusReqReader instanceof WALNode)) {
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot recover from non-empty region progress without WAL access: %s",
+ this, recoveryRegionProgress));
+ }
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(recoveryRegionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ return new ReplayLocateDecision(
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ progressSource + " (" + replayTarget.getDetail() + ")");
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot initialize replay start from region progress %s: %s",
+ this, recoveryRegionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private boolean shouldUseConsumerRegionProgressHint(
+ final RegionProgress regionProgress, final RegionProgress committedRegionProgress) {
+ if (Objects.isNull(regionProgress) || regionProgress.getWriterPositions().isEmpty()) {
+ return false;
+ }
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return true;
+ }
+ for (final Map.Entry<WriterId, WriterProgress> entry :
+ regionProgress.getWriterPositions().entrySet()) {
+ if (Objects.isNull(entry.getKey()) || Objects.isNull(entry.getValue())) {
+ continue;
+ }
+ final WriterProgress committedWriterProgress =
+ committedRegionProgress.getWriterPositions().get(entry.getKey());
+ if (Objects.isNull(committedWriterProgress)
+ || compareWriterProgress(entry.getValue(), committedWriterProgress) > 0) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private RegionProgress mergeRecoveryRegionProgress(
+ final RegionProgress committedRegionProgress, final RegionProgress consumerRegionProgress) {
+ if (Objects.isNull(committedRegionProgress)
+ || committedRegionProgress.getWriterPositions().isEmpty()) {
+ return consumerRegionProgress;
+ }
+ if (Objects.isNull(consumerRegionProgress)
+ || consumerRegionProgress.getWriterPositions().isEmpty()) {
+ return committedRegionProgress;
+ }
+
+ final Map<WriterId, WriterProgress> mergedWriterProgress = new LinkedHashMap<>();
+ committedRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ mergedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ consumerRegionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.isNull(writerId) || Objects.isNull(writerProgress)) {
+ return;
+ }
+ mergedWriterProgress.merge(
+ writerId,
+ writerProgress,
+ (committedWriterProgress, consumerWriterProgress) ->
+ compareWriterProgress(consumerWriterProgress, committedWriterProgress) > 0
+ ? consumerWriterProgress
+ : committedWriterProgress);
+ });
+ return new RegionProgress(mergedWriterProgress);
+ }
+
+ protected RegionProgress resolveCommittedRegionProgressForInit() {
+ commitManager.getOrCreateState(brokerId, topicName, consensusGroupId);
+ final RegionProgress latestCommittedRegionProgress =
+ commitManager.getCommittedRegionProgress(brokerId, topicName, consensusGroupId);
+ if (Objects.nonNull(latestCommittedRegionProgress)
+ && !latestCommittedRegionProgress.getWriterPositions().isEmpty()) {
+ return latestCommittedRegionProgress;
+ }
+ return Objects.nonNull(fallbackCommittedRegionProgress)
+ && !fallbackCommittedRegionProgress.getWriterPositions().isEmpty()
+ ? fallbackCommittedRegionProgress
+ : null;
+ }
+
+ private void installRecoveryWriterProgress(final RegionProgress regionProgress) {
+ recoveryWriterProgressByWriter.clear();
+ recoveryWriterProgressByWriter.putAll(regionProgress.getWriterPositions());
+ regionProgress
+ .getWriterPositions()
+ .keySet()
+ .forEach(writerId -> trackWriterLane(writerId.getNodeId()));
+ }
+
+ private void clearRecoveryWriterProgress() {
+ recoveryWriterProgressByWriter.clear();
+ }
+
+ private boolean shouldSkipForRecoveryProgress(final IndexedConsensusRequest request) {
+ if (recoveryWriterProgressByWriter.isEmpty()) {
+ return false;
+ }
+ return isRequestCoveredByRegionProgress(request, recoveryWriterProgressByWriter, true);
+ }
+
+ private boolean hasComparableWriterProgress(final IndexedConsensusRequest request) {
+ return request.getNodeId() >= 0
+ && request.getPhysicalTime() > 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private WriterId toWriterId(final IndexedConsensusRequest request) {
+ return new WriterId(consensusGroupId.toString(), request.getNodeId());
+ }
+
+ private WriterProgress toWriterProgress(final IndexedConsensusRequest request) {
+ return new WriterProgress(request.getPhysicalTime(), request.getProgressLocalSeq());
+ }
+
+ private boolean isRequestCoveredByRegionProgress(
+ final IndexedConsensusRequest request,
+ final Map<WriterId, WriterProgress> regionProgressByWriter,
+ final boolean seekAfter) {
+ if (!hasComparableWriterProgress(request)) {
+ return false;
+ }
+ final WriterProgress committedProgress = regionProgressByWriter.get(toWriterId(request));
+ if (Objects.isNull(committedProgress)) {
+ return false;
+ }
+ final int cmp = compareWriterProgress(toWriterProgress(request), committedProgress);
+ return seekAfter ? cmp <= 0 : cmp < 0;
+ }
+
+ private WriterProgress decrementWriterProgress(final WriterProgress writerProgress) {
+ return new WriterProgress(
+ writerProgress.getPhysicalTime(),
+ writerProgress.getLocalSeq() > 0L
+ ? writerProgress.getLocalSeq() - 1L
+ : BEFORE_FIRST_LOCAL_SEQ);
+ }
+
+ protected ReplayLocateDecision scanReplayStartForRequests(
+ final Iterable<IndexedConsensusRequest> requests,
+ final RegionProgress regionProgress,
+ final boolean seekAfter) {
+ final Map<WriterId, WriterProgress> requestedWriterProgress = new LinkedHashMap<>();
+ if (Objects.nonNull(regionProgress)) {
+ regionProgress
+ .getWriterPositions()
+ .forEach(
+ (writerId, writerProgress) -> {
+ if (Objects.nonNull(writerId) && Objects.nonNull(writerProgress)) {
+ requestedWriterProgress.put(writerId, writerProgress);
+ }
+ });
+ }
+ final Map<WriterId, WriterProgress> effectiveRecoveryWriterProgress =
+ new LinkedHashMap<>(requestedWriterProgress);
+ final Set<WriterId> exactVisibleWriterIds = new LinkedHashSet<>();
+ Long firstUncoveredReplayableSearchIndex = null;
+ boolean sawBlockingNonReplayableUncovered = false;
+
+ for (final IndexedConsensusRequest request : requests) {
+ if (!hasComparableWriterProgress(request)) {
+ continue;
+ }
+
+ final WriterId writerId = toWriterId(request);
+ final WriterProgress requestProgress = toWriterProgress(request);
+ final WriterProgress storedWriterProgress = requestedWriterProgress.get(writerId);
+ if (!seekAfter
+ && Objects.nonNull(storedWriterProgress)
+ && compareWriterProgress(requestProgress, storedWriterProgress) == 0) {
+ exactVisibleWriterIds.add(writerId);
+ }
+
+ if (isRequestCoveredByRegionProgress(request, requestedWriterProgress, seekAfter)) {
+ continue;
+ }
+
+ if (request.getSearchIndex() >= 0) {
+ if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ firstUncoveredReplayableSearchIndex = request.getSearchIndex();
+ }
+ } else if (Objects.isNull(firstUncoveredReplayableSearchIndex)) {
+ sawBlockingNonReplayableUncovered = true;
+ }
+ }
+
+ if (!seekAfter && !exactVisibleWriterIds.isEmpty()) {
+ for (final WriterId writerId : exactVisibleWriterIds) {
+ final WriterProgress writerProgress = requestedWriterProgress.get(writerId);
+ if (Objects.nonNull(writerProgress)) {
+ effectiveRecoveryWriterProgress.put(writerId, decrementWriterProgress(writerProgress));
+ }
+ }
+ }
+ final RegionProgress effectiveRecoveryRegionProgress =
+ new RegionProgress(effectiveRecoveryWriterProgress);
+
+ if (sawBlockingNonReplayableUncovered) {
+ return ReplayLocateDecision.locateMiss(
+ effectiveRecoveryRegionProgress,
+ "uncovered non-replayable WAL records appear before the first local replayable record");
+ }
+ if (Objects.nonNull(firstUncoveredReplayableSearchIndex)) {
+ return ReplayLocateDecision.found(
+ firstUncoveredReplayableSearchIndex,
+ effectiveRecoveryRegionProgress,
+ "resolved first uncovered replayable WAL record");
+ }
+ return ReplayLocateDecision.atEnd(
+ consensusReqReader.getCurrentSearchIndex(),
+ computeTailRegionProgress(),
+ "all locally replayable WAL records are already covered");
+ }
+
+ protected ReplayLocateDecision locateReplayStartForRegionProgress(
+ final RegionProgress regionProgress, final boolean seekAfter) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "WAL access is unavailable for region-level replay lookup");
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final List<IndexedConsensusRequest> replayRequests = new ArrayList<>();
+ try (final ProgressWALIterator iterator = new ProgressWALIterator(walNode, Long.MIN_VALUE)) {
+ while (iterator.hasNext()) {
+ replayRequests.add(iterator.next());
+ }
+ if (iterator.hasIncompleteScan()) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress,
+ "replay lookup did not complete: " + iterator.getIncompleteScanDetail());
+ }
+ return scanReplayStartForRequests(replayRequests, regionProgress, seekAfter);
+ } catch (final IOException e) {
+ return ReplayLocateDecision.locateMiss(
+ regionProgress, "failed to close replay lookup iterator: " + e.getMessage());
+ }
+ }
+
+ private boolean shouldTrackFollowerProgressForDedup(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() < 0
+ && request.getNodeId() >= 0
+ && request.getProgressLocalSeq() >= 0;
+ }
+
+ private boolean shouldSkipForMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return false;
+ }
+ final Long materializedLocalSeq =
+ materializedFollowerProgressByWriter.get(new WriterLaneId(request.getNodeId()));
+ return Objects.nonNull(materializedLocalSeq)
+ && request.getProgressLocalSeq() <= materializedLocalSeq;
+ }
+
+ private void markMaterializedFollowerProgress(final IndexedConsensusRequest request) {
+ if (!shouldTrackFollowerProgressForDedup(request)) {
+ return;
+ }
+ materializedFollowerProgressByWriter.merge(
+ new WriterLaneId(request.getNodeId()), request.getProgressLocalSeq(), Math::max);
+ }
+
+ private int compareWriterProgress(
+ final WriterProgress leftProgress, final WriterProgress rightProgress) {
+ int cmp = Long.compare(leftProgress.getPhysicalTime(), rightProgress.getPhysicalTime());
+ if (cmp != 0) {
+ return cmp;
+ }
+ return Long.compare(leftProgress.getLocalSeq(), rightProgress.getLocalSeq());
+ }
+
+ private WriterLaneState trackWriterLane(final int writerNodeId) {
+ return writerLanes.computeIfAbsent(
+ new WriterLaneId(writerNodeId), ignored -> new WriterLaneState());
+ }
+
+ private void refreshWriterLaneSafeFrontiers() {
+ final Map<WriterSafeFrontierTracker.WriterIdentity, Long> safePts =
+ serverImpl.getWriterSafeFrontierTracker().snapshotEffectiveSafePts();
+ for (final Map.Entry<WriterSafeFrontierTracker.WriterIdentity, Long> entry :
+ safePts.entrySet()) {
+ final WriterLaneState laneState = trackWriterLane(entry.getKey().getWriterNodeId());
+ laneState.effectiveSafePt = Math.max(laneState.effectiveSafePt, entry.getValue());
+ }
+ }
+
+ private <T extends LaneBufferedEntry> PriorityQueue<LaneFrontier> buildLaneFrontiers(
+ final Map<WriterLaneId, ?> laneEntriesByLane, final Function<WriterLaneId, T> headSupplier) {
+ refreshWriterLaneSafeFrontiers();
+ final PriorityQueue<LaneFrontier> frontiers = new PriorityQueue<>();
+ final boolean useActiveWriterBarriers = shouldUseActiveWriterBarriers();
+ final Set<WriterLaneId> laneIds = ConcurrentHashMap.newKeySet();
+ final Set<Integer> seenActiveWriterNodeIds = ConcurrentHashMap.newKeySet();
+ laneIds.addAll(writerLanes.keySet());
+ laneIds.addAll(laneEntriesByLane.keySet());
+ for (final WriterLaneId laneId : laneIds) {
+ final WriterLaneState laneState = writerLanes.get(laneId);
+ if (Objects.nonNull(laneState) && laneState.closed) {
+ continue;
+ }
+ final T head = headSupplier.apply(laneId);
+ if (Objects.nonNull(head)) {
+ if (isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ }
+ frontiers.add(LaneFrontier.forHead(laneId, head));
+ continue;
+ }
+ if (Objects.nonNull(laneState)
+ && laneState.effectiveSafePt > 0
+ && useActiveWriterBarriers
+ && isLaneRuntimeActive(laneId)) {
+ seenActiveWriterNodeIds.add(laneId.writerNodeId);
+ frontiers.add(LaneFrontier.forBarrier(laneId, laneState.effectiveSafePt));
+ }
+ }
+ if (useActiveWriterBarriers) {
+ for (final Integer activeWriterNodeId : activeWriterNodeIds) {
+ if (!seenActiveWriterNodeIds.contains(activeWriterNodeId)) {
+ frontiers.add(
+ LaneFrontier.forBarrier(new WriterLaneId(activeWriterNodeId), Long.MIN_VALUE));
+ break;
+ }
+ }
+ }
+ return frontiers;
+ }
+
+ private boolean shouldUseActiveWriterBarriers() {
+ return !TopicConstant.ORDER_MODE_PER_WRITER_VALUE.equals(orderMode);
+ }
+
+ private void bufferRealtimeEntry(final PreparedEntry entry) {
+ final WriterLaneId laneId = new WriterLaneId(entry.writerNodeId);
+ realtimeEntriesByLane
+ .computeIfAbsent(laneId, ignored -> new TreeMap<>())
+ .put(entry.localSeq, entry);
+ }
+
+ private PreparedEntry peekRealtimeEntry(final WriterLaneId laneId) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries) || laneEntries.isEmpty()) {
+ return null;
+ }
+ final Map.Entry<Long, PreparedEntry> firstEntry = laneEntries.firstEntry();
+ return Objects.nonNull(firstEntry) ? firstEntry.getValue() : null;
+ }
+
+ private void removeRealtimeEntry(final WriterLaneId laneId, final long localSeq) {
+ final NavigableMap<Long, PreparedEntry> laneEntries = realtimeEntriesByLane.get(laneId);
+ if (Objects.isNull(laneEntries)) {
+ return;
+ }
+ laneEntries.remove(localSeq);
+ if (laneEntries.isEmpty()) {
+ realtimeEntriesByLane.remove(laneId);
+ }
+ }
+
+ private PriorityQueue<LaneFrontier> buildRealtimeLaneFrontiers() {
+ return buildLaneFrontiers(realtimeEntriesByLane, this::peekRealtimeEntry);
+ }
+
+ private SubscriptionEvent pollInternal(final String consumerId) {
+ final long size = prefetchingQueue.size();
+ if (size == 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: prefetching queue is empty for consumerId={}, "
+ + "pendingEntriesSize={}, nextExpected={}, isClosed={}, prefetchInitialized={}, subtaskScheduled={}",
+ this,
+ consumerId,
+ pendingEntries.size(),
+ nextExpectedSearchIndex.get(),
+ isClosed,
+ prefetchInitialized,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ return null;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: polling, queue size={}, consumerId={}",
+ this,
+ size,
+ consumerId);
+ long count = 0;
+
+ SubscriptionEvent event;
+ try {
+ while (count++ < size
+ && Objects.nonNull(
+ event =
+ prefetchingQueue.poll(
+ SubscriptionConfig.getInstance().getSubscriptionPollMaxBlockingTimeMs(),
+ TimeUnit.MILLISECONDS))) {
+ // Metadata events (currently WATERMARK) are fire-and-forget:
+ // skip inFlightEvents tracking so they are not recycled and re-delivered indefinitely.
+ if (event.getCurrentResponse().getResponseType()
+ == SubscriptionPollResponseType.WATERMARK.getType()) {
+ return event;
+ }
+
+ if (event.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll committed event {} (broken invariant), remove it",
+ this,
+ event);
+ continue;
+ }
+
+ if (!event.pollable()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {} poll non-pollable event {} (broken invariant), nack it",
+ this,
+ event);
+ event.nack();
+ continue;
+ }
+
+ // Mark as polled before updating inFlightEvents
+ event.recordLastPolledTimestamp();
+ inFlightEvents.put(new Pair<>(consumerId, event.getCommitContext()), event);
+ event.recordLastPolledConsumerId(consumerId);
+ return event;
+ }
+ } catch (final InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOGGER.warn("ConsensusPrefetchingQueue {} interrupted while polling", this, e);
+ }
+
+ return null;
+ }
+
+ public SubscriptionEvent pollTablets(
+ final String consumerId, final SubscriptionCommitContext commitContext, final int offset) {
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return null;
+ }
+ final SubscriptionEvent event = inFlightEvents.get(new Pair<>(consumerId, commitContext));
+ if (Objects.isNull(event)) {
+ if (isCommitContextOutdated(commitContext)) {
+ return generateOutdatedErrorResponse();
+ }
+ return generateErrorResponse(
+ String.format(
+ "ConsensusPrefetchingQueue %s: no in-flight event for consumer %s, commit context %s",
+ this, consumerId, commitContext));
+ }
+ return event;
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ // ======================== Prefetch Round Drive ========================
+
+ private static final long WAL_GAP_RETRY_SLEEP_MS = 10L;
+ private static final long WAL_GAP_WAIT_LOG_INTERVAL_MS = 5_000L;
+
+ private static final long PREFETCH_STATS_LOG_INTERVAL_MS = 5_000L;
+
+ public PrefetchRoundResult drivePrefetchOnce() {
+ if (applyPendingSeekRequestIfNecessary()) {
+ return closeRequested ? PrefetchRoundResult.dormant() : PrefetchRoundResult.rescheduleNow();
+ }
+
+ acquireReadLock();
+ try {
+ if (isClosed || closeRequested || !prefetchInitialized) {
+ return PrefetchRoundResult.dormant();
+ }
+
+ logPeriodicStatsIfNecessary();
+
+ final long currentSeekGeneration = seekGeneration.get();
+ if (currentSeekGeneration != observedSeekGeneration) {
+ resetRoundStateForSeek(currentSeekGeneration);
+ }
+
+ applyPendingSubscriptionWalReset(observedSeekGeneration);
+ recycleInFlightEvents();
+
+ if (!isActive || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return computeIdleRoundResult();
+ }
+
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+ final int batchMaxDelayMs = config.getSubscriptionConsensusBatchMaxDelayInMs();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+
+ final List<IndexedConsensusRequest> batch = drainPendingEntries(maxWalEntries);
+ if (!batch.isEmpty()) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: drained {} entries from pendingEntries, "
+ + "first searchIndex={}, last searchIndex={}, nextExpected={}, "
+ + "prefetchingQueueSize={}",
+ this,
+ batch.size(),
+ batch.get(0).getSearchIndex(),
+ batch.get(batch.size() - 1).getSearchIndex(),
+ nextExpectedSearchIndex.get(),
+ prefetchingQueue.size());
+
+ final boolean batchAccepted =
+ accumulateFromPending(
+ batch, lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes);
+ if (!batchAccepted) {
+ if (pendingWalGapRetryRequested) {
+ // Once a drained batch hits an unresolved WAL gap, the affected suffix falls back to
+ // the WAL path on later rounds instead of being requeued into the bounded pending path.
+ return PrefetchRoundResult.rescheduleAfter(WAL_GAP_RETRY_SLEEP_MS);
+ }
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ }
+
+ if (batch.isEmpty() && lingerBatch.isEmpty()) {
+ tryCatchUpFromWAL(observedSeekGeneration);
+ }
+
+ if (!drainBufferedRealtimeLanes(
+ lingerBatch, observedSeekGeneration, maxTablets, maxBatchBytes)) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerElapsedMs = System.currentTimeMillis() - lingerBatch.firstTabletTimeMs;
+ if (lingerElapsedMs >= batchMaxDelayMs) {
+ if (seekGeneration.get() != observedSeekGeneration) {
+ resetRoundStateForSeek(seekGeneration.get());
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: time-based flush, {} tablets lingered for {}ms "
+ + "(threshold={}ms)",
+ this,
+ lingerBatch.tablets.size(),
+ lingerElapsedMs,
+ batchMaxDelayMs);
+ flushBatch(lingerBatch, observedSeekGeneration);
+ }
+ }
+
+ maybeInjectWatermark();
+ return computeIdleRoundResult();
+ } catch (final Throwable fatal) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: prefetch round failed " + "(type={}, message={})",
+ this,
+ fatal.getClass().getName(),
+ fatal.getMessage(),
+ fatal);
+ if (fatal instanceof VirtualMachineError) {
+ markClosed();
+ return PrefetchRoundResult.dormant();
+ }
+ return PrefetchRoundResult.rescheduleAfter(100L);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private void logPeriodicStatsIfNecessary() {
+ final long nowMs = System.currentTimeMillis();
+ if (nowMs - lastStatsLogTimeMs < PREFETCH_STATS_LOG_INTERVAL_MS) {
+ return;
+ }
+
+ final long currentPendingAcceptedEntries = pendingPathAcceptedEntries.get();
+ final long currentWalAcceptedEntries = walPathAcceptedEntries.get();
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: periodic stats, lag={}, pendingDelta={}, walDelta={}, "
+ + "pendingTotal={}, walTotal={}, pendingQueueSize={}, prefetchingQueueSize={}, "
+ + "inFlightEventsSize={}, realtimeLaneCount={}, walHasNext={}, isActive={}, subtaskScheduled={}",
+ this,
+ getLag(),
+ currentPendingAcceptedEntries - lastPendingAcceptedEntries,
+ currentWalAcceptedEntries - lastWalAcceptedEntries,
+ currentPendingAcceptedEntries,
+ currentWalAcceptedEntries,
+ pendingEntries.size(),
+ prefetchingQueue.size(),
+ inFlightEvents.size(),
+ realtimeEntriesByLane.size(),
+ hasReadableWalEntries(),
+ isActive,
+ Objects.nonNull(prefetchSubtask) && prefetchSubtask.isScheduledOrRunning());
+ lastStatsLogTimeMs = nowMs;
+ lastPendingAcceptedEntries = currentPendingAcceptedEntries;
+ lastWalAcceptedEntries = currentWalAcceptedEntries;
+ }
+
+ private void resetRoundStateForSeek(final long newSeekGeneration) {
+ restorePendingSubscriptionWalCursor(newSeekGeneration);
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = newSeekGeneration;
+ }
+
+ private List<IndexedConsensusRequest> drainPendingEntries(final int maxWalEntries) {
+ final List<IndexedConsensusRequest> batch = new ArrayList<>();
+ IndexedConsensusRequest next;
+ while (batch.size() < maxWalEntries && (next = pendingEntries.poll()) != null) {
+ batch.add(next);
+ }
+ return batch;
+ }
+
+ private PrefetchRoundResult computeIdleRoundResult() {
+ if (isClosed || !prefetchInitialized || !isActive) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return PrefetchRoundResult.dormant();
+ }
+ if (hasImmediatePrefetchableWork()) {
+ return PrefetchRoundResult.rescheduleNow();
+ }
+ long delayMs = Long.MAX_VALUE;
+ if (hasHistoricalWalLag()) {
+ delayMs = Math.min(delayMs, WAL_GAP_RETRY_SLEEP_MS);
+ }
+ if (!lingerBatch.isEmpty() && lingerBatch.firstTabletTimeMs > 0L) {
+ final long lingerDelayMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusBatchMaxDelayInMs()
+ - (System.currentTimeMillis() - lingerBatch.firstTabletTimeMs);
+ delayMs = Math.min(delayMs, Math.max(1L, lingerDelayMs));
+ }
+
+ final long watermarkDelayMs = computeWatermarkDelayMs();
+ if (watermarkDelayMs > 0L) {
+ delayMs = Math.min(delayMs, watermarkDelayMs);
+ }
+
+ if (!inFlightEvents.isEmpty()) {
+ delayMs =
+ Math.min(
+ delayMs,
+ SubscriptionConfig.getInstance().getSubscriptionRecycleUncommittedEventIntervalMs());
+ }
+
+ return delayMs == Long.MAX_VALUE
+ ? PrefetchRoundResult.dormant()
+ : PrefetchRoundResult.rescheduleAfter(delayMs);
+ }
+
+ private long computeWatermarkDelayMs() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return -1L;
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0L) {
+ return -1L;
+ }
+ if (lastWatermarkEmitTimeMs == 0L) {
+ return 1L;
+ }
+ final long elapsedMs = System.currentTimeMillis() - lastWatermarkEmitTimeMs;
+ return elapsedMs >= intervalMs ? 1L : Math.max(1L, intervalMs - elapsedMs);
+ }
+
+ private boolean hasImmediatePrefetchableWork() {
+ return !pendingEntries.isEmpty() || !realtimeEntriesByLane.isEmpty() || hasReadableWalEntries();
+ }
+
+ private boolean hasHistoricalWalLag() {
+ return nextExpectedSearchIndex.get() < consensusReqReader.getCurrentSearchIndex();
+ }
+
+ /**
+ * Accumulates tablets from pending entries into the linger buffer. When pending replay outruns
+ * the local WAL reader, this method backfills the local-index gap from WAL before continuing.
+ *
+ * @return false if the batch became stale because seek generation changed while flushing
+ */
+ private static boolean hasLocalSearchIndex(final IndexedConsensusRequest request) {
+ return request.getSearchIndex() >= 0;
+ }
+
+ private boolean isBeforeLocalCursor(final IndexedConsensusRequest request) {
+ return hasLocalSearchIndex(request) && request.getSearchIndex() < nextExpectedSearchIndex.get();
+ }
+
+ private void advanceLocalCursorIfPresent(final IndexedConsensusRequest request) {
+ if (hasLocalSearchIndex(request)) {
+ nextExpectedSearchIndex.set(request.getSearchIndex() + 1);
+ }
+ }
+
+ private boolean appendRealtimeRequest(
+ final IndexedConsensusRequest request,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean fromPending) {
+ final PreparedEntry preparedEntry = prepareEntry(request);
+ if (Objects.isNull(preparedEntry)) {
+ return true;
+ }
+ if (!appendPreparedEntryViaRealtimeLane(
+ batchState, preparedEntry, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+ if (fromPending) {
+ markAcceptedFromPending();
+ } else {
+ markAcceptedFromWal();
+ }
+ return true;
+ }
+
+ private boolean accumulateFromPending(
+ final List<IndexedConsensusRequest> batch,
+ final DeliveryBatchState lingerBatch,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+
+ int processedCount = 0;
+ int skippedCount = 0;
+
+ for (int index = 0; index < batch.size(); index++) {
+ final IndexedConsensusRequest request = batch.get(index);
+ final long searchIndex = request.getSearchIndex();
+
+ // Only local-indexed requests participate in the internal WAL read cursor.
+ final long expected = nextExpectedSearchIndex.get();
+ if (hasLocalSearchIndex(request) && searchIndex > expected) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: gap detected, expected={}, got={}. "
+ + "Filling {} entries from WAL.",
+ this,
+ expected,
+ searchIndex,
+ searchIndex - expected);
+ if (!fillGapFromWAL(
+ expected,
+ searchIndex,
+ lingerBatch,
+ expectedSeekGeneration,
+ maxTablets,
+ maxBatchBytes)) {
+ return false;
+ }
+ }
+
+ if (isBeforeLocalCursor(request)) {
+ skippedCount++;
+ continue;
+ }
+
+ if (shouldSkipForRecoveryProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(request)) {
+ skippedCount++;
+ advanceLocalCursorIfPresent(request);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ request, lingerBatch, expectedSeekGeneration, maxTablets, maxBatchBytes, true)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(request);
+ processedCount++;
+ advanceLocalCursorIfPresent(request);
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: accumulate complete, batchSize={}, processed={}, "
+ + "skipped={}, lingerTablets={}, nextExpected={}",
+ this,
+ batch.size(),
+ processedCount,
+ skippedCount,
+ lingerBatch.tablets.size(),
+ nextExpectedSearchIndex.get());
+
+ return true;
+ }
+
+ /**
+ * Fills a gap in the pending queue by reading entries from WAL so the internal local replay
+ * cursor stays contiguous even when pending delivery jumps ahead of the WAL iterator.
+ *
+ * <p>Temporary WAL visibility lag is treated as a normal back-pressure condition: once a drained
+ * pending batch encounters an unresolved local-index gap, the queue backs off and lets the
+ * affected suffix fall back to the WAL path on later rounds. This keeps replay contiguous without
+ * requeueing the drained batch back into the bounded pending queue.
+ *
+ * @return false if gap fill had to stop because the current batch became stale or the queue was
+ * interrupted/closed
+ */
+ private boolean fillGapFromWAL(
+ final long fromIndex,
+ final long toIndex,
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ pendingWalGapRetryRequested = false;
+ resetSubscriptionWALPosition(fromIndex);
+ if (seekGeneration.get() != expectedSeekGeneration || isClosed) {
+ return false;
+ }
+ if (!pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, Integer.MAX_VALUE, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final long nextExpected = nextExpectedSearchIndex.get();
+ if (nextExpected >= toIndex) {
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ return true;
+ }
+
+ final long nowMs = System.currentTimeMillis();
+ if (walGapWaitStartTimeMs == 0L) {
+ walGapWaitStartTimeMs = nowMs;
+ }
+ if (lastWalGapWaitLogTimeMs == 0L
+ || nowMs - lastWalGapWaitLogTimeMs >= WAL_GAP_WAIT_LOG_INTERVAL_MS) {
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: waiting {}ms for WAL gap [{}, {}) to become visible, "
+ + "currentNextExpected={}, currentWalIndex={}, seekGeneration={}",
+ this,
+ nowMs - walGapWaitStartTimeMs,
+ nextExpected,
+ toIndex,
+ nextExpected,
+ consensusReqReader.getCurrentSearchIndex(),
+ expectedSeekGeneration);
+ lastWalGapWaitLogTimeMs = nowMs;
+ }
+ onWalGapRetryScheduled();
+ pendingWalGapRetryRequested = true;
+ return false;
+ }
+
+ /**
+ * Try catch-up from WAL when the pending queue was empty. This handles cold-start or scenarios
+ * where the subscription started after data was already written.
+ */
+ private void tryCatchUpFromWAL(final long expectedSeekGeneration) {
+ final SubscriptionConfig config = SubscriptionConfig.getInstance();
+ final int maxTablets = config.getSubscriptionConsensusBatchMaxTabletCount();
+ final long maxBatchBytes = config.getSubscriptionConsensusBatchMaxSizeInBytes();
+ final int maxWalEntries = config.getSubscriptionConsensusBatchMaxWalEntries();
+
+ final DeliveryBatchState batchState = new DeliveryBatchState();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ final boolean accepted =
+ pumpFromSubscriptionWAL(
+ batchState, expectedSeekGeneration, maxWalEntries, maxTablets, maxBatchBytes);
+ if (!accepted) {
+ return;
+ }
+
+ if (!batchState.isEmpty()) {
+ flushBatch(batchState, expectedSeekGeneration);
+ }
+ }
+
+ private boolean pumpFromSubscriptionWAL(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxWalEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return true;
+ }
+
+ subscriptionWALIterator.refresh();
+ ensureSubscriptionWalReadable();
+
+ int entriesRead = 0;
+ while (entriesRead < maxWalEntries
+ && subscriptionWALIterator.hasNext()
+ && prefetchingQueue.size() < MAX_PREFETCHING_QUEUE_SIZE) {
+ try {
+ final IndexedConsensusRequest walEntry = subscriptionWALIterator.next();
+ entriesRead++;
+
+ if (isBeforeLocalCursor(walEntry)) {
+ continue;
+ }
+ if (shouldSkipForRecoveryProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+ if (shouldSkipForMaterializedFollowerProgress(walEntry)) {
+ advanceLocalCursorIfPresent(walEntry);
+ continue;
+ }
+
+ if (!appendRealtimeRequest(
+ walEntry, batchState, expectedSeekGeneration, maxTablets, maxBatchBytes, false)) {
+ return false;
+ }
+ markMaterializedFollowerProgress(walEntry);
+ advanceLocalCursorIfPresent(walEntry);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error reading subscription WAL", this, e);
+ break;
+ }
+ }
+
+ if (entriesRead > 0) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL read {} entries, nextExpectedSearchIndex={}",
+ this,
+ entriesRead,
+ nextExpectedSearchIndex.get());
+ }
+ return true;
+ }
+
+ private void ensureSubscriptionWalReadable() {
+ if (Objects.isNull(subscriptionWALIterator)
+ || subscriptionWALIterator.hasNext()
+ || !(consensusReqReader instanceof WALNode)) {
+ return;
+ }
+
+ final long currentWalIndex = consensusReqReader.getCurrentSearchIndex();
+ if (nextExpectedSearchIndex.get() > currentWalIndex) {
+ return;
+ }
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: subscription WAL exhausted at {} while current WAL is {}. "
+ + "Rolling WAL file to expose current-file entries.",
+ this,
+ nextExpectedSearchIndex.get(),
+ currentWalIndex);
+ ((WALNode) consensusReqReader).rollWALFile();
+ resetSubscriptionWALPosition(nextExpectedSearchIndex.get());
+ if (Objects.nonNull(subscriptionWALIterator)) {
+ subscriptionWALIterator.refresh();
+ }
+ }
+
+ private void resetSubscriptionWALPosition(final long startSearchIndex) {
+ closeSubscriptionWALIterator();
+ subscriptionWALIterator = createSubscriptionWALIterator(startSearchIndex);
+ }
+
+ protected ProgressWALIterator createSubscriptionWALIterator(final long startSearchIndex) {
+ if (consensusReqReader instanceof WALNode) {
+ return new ProgressWALIterator((WALNode) consensusReqReader, startSearchIndex);
+ }
+ return null;
+ }
+
+ protected void onWalGapRetryScheduled() {}
+
+ private boolean hasReadableWalEntries() {
+ return Objects.nonNull(subscriptionWALIterator) && subscriptionWALIterator.hasNext();
+ }
+
+ private void requestSubscriptionWalReset(
+ final long targetSearchIndex, final long seekGenerationValue) {
+ pendingSubscriptionWalResetSearchIndex = targetSearchIndex;
+ pendingSubscriptionWalResetGeneration = seekGenerationValue;
+ }
+
+ private void applyPendingSubscriptionWalReset(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ resetSubscriptionWALPosition(pendingSubscriptionWalResetSearchIndex);
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ }
+
+ private void restorePendingSubscriptionWalCursor(final long observedSeekGeneration) {
+ if (pendingSubscriptionWalResetGeneration != observedSeekGeneration
+ || pendingSubscriptionWalResetSearchIndex == Long.MIN_VALUE) {
+ return;
+ }
+ // A seek can land in the middle of a prefetch iteration. Restore the local cursor to the
+ // pending seek target before resuming under the new generation so stale in-flight work does
+ // not permanently advance the historical replay frontier.
+ nextExpectedSearchIndex.set(pendingSubscriptionWalResetSearchIndex);
+ }
+
+ private void closeSubscriptionWALIterator() {
+ if (Objects.isNull(subscriptionWALIterator)) {
+ return;
+ }
+ try {
+ subscriptionWALIterator.close();
+ } catch (final IOException e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error closing subscription WAL iterator", this, e);
+ } finally {
+ subscriptionWALIterator = null;
+ }
+ }
+
+ /**
+ * Deserializes the IConsensusRequest entries within an IndexedConsensusRequest to produce an
+ * InsertNode. WAL entries are typically stored as IoTConsensusRequest (serialized ByteBuffers),
+ * and a single logical write may be split across multiple fragments (SearchNode). This method
+ * handles both cases.
+ *
+ * <p>The deserialization follows the same pattern as {@code
+ * DataRegionStateMachine.grabPlanNode()}.
+ */
+ private InsertNode deserializeToInsertNode(final IndexedConsensusRequest indexedRequest) {
+ final List<SearchNode> searchNodes = new ArrayList<>();
+ PlanNode nonSearchNode = null;
+
+ for (final IConsensusRequest req : indexedRequest.getRequests()) {
+ PlanNode planNode;
+ try {
+ if (req instanceof IoTConsensusRequest) {
+ // WAL entries read from file are wrapped as IoTConsensusRequest (ByteBuffer)
+ planNode = WALEntry.deserializeForConsensus(req.serializeToByteBuffer());
+ } else if (req instanceof InsertNode) {
+ // In-memory entries (not yet flushed to WAL file) may already be PlanNode
+ planNode = (PlanNode) req;
+ } else {
+ // ByteBufferConsensusRequest or unknown
+ planNode = PlanNodeType.deserialize(req.serializeToByteBuffer());
+ }
+ } catch (final Exception e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to deserialize IConsensusRequest "
+ + "(type={}) in searchIndex={}: {}",
+ this,
+ req.getClass().getSimpleName(),
+ indexedRequest.getSearchIndex(),
+ e.getMessage(),
+ e);
+ continue;
+ }
+
+ if (planNode instanceof SearchNode) {
+ final SearchNode searchNode = (SearchNode) planNode;
+ searchNode.setSearchIndex(indexedRequest.getSearchIndex());
+ if (indexedRequest.getSyncIndex() >= 0) {
+ searchNode.setSyncIndex(indexedRequest.getSyncIndex());
+ }
+ if (indexedRequest.getPhysicalTime() > 0) {
+ searchNode.setPhysicalTime(indexedRequest.getPhysicalTime());
+ }
+ if (indexedRequest.getNodeId() >= 0) {
+ searchNode.setNodeId(indexedRequest.getNodeId());
+ }
+ searchNodes.add(searchNode);
+ } else {
+ nonSearchNode = planNode;
+ }
+ }
+
+ // Merge split SearchNode fragments (same pattern as DataRegionStateMachine.grabPlanNode)
+ if (!searchNodes.isEmpty()) {
+ final PlanNode merged = searchNodes.get(0).merge(searchNodes);
+ if (merged instanceof InsertNode) {
+ final InsertNode mergedInsert = (InsertNode) merged;
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: deserialized merged InsertNode for searchIndex={}, "
+ + "type={}, deviceId={}, searchNodeCount={}",
+ this,
+ indexedRequest.getSearchIndex(),
+ mergedInsert.getType(),
+ ConsensusLogToTabletConverter.safeDeviceIdForLog(mergedInsert),
+ searchNodes.size());
+
+ return mergedInsert;
+ }
+ }
+
+ if (nonSearchNode != null) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: searchIndex={} contains non-InsertNode PlanNode: {}",
+ this,
+ indexedRequest.getSearchIndex(),
+ nonSearchNode.getClass().getSimpleName());
+ }
+
+ return null;
+ }
+
+ private PreparedEntry prepareEntry(final IndexedConsensusRequest indexedRequest) {
+ final InsertNode insertNode = deserializeToInsertNode(indexedRequest);
+ if (Objects.isNull(insertNode)) {
+ return null;
+ }
+
+ final long localSeq =
+ indexedRequest.getProgressLocalSeq() >= 0
+ ? indexedRequest.getProgressLocalSeq()
+ : indexedRequest.getSearchIndex();
+ final long searchIndex = indexedRequest.getSearchIndex();
+ final long physicalTime =
+ indexedRequest.getPhysicalTime() > 0
+ ? indexedRequest.getPhysicalTime()
+ : insertNode.getPhysicalTime();
+ final int writerNodeId =
+ indexedRequest.getNodeId() >= 0 ? indexedRequest.getNodeId() : insertNode.getNodeId();
+
+ trackWriterLane(writerNodeId);
+ final long maxTs = extractMaxTime(insertNode);
+ if (maxTs > maxObservedTimestamp) {
+ maxObservedTimestamp = maxTs;
+ }
+ final List<Tablet> tablets = converter.convert(insertNode);
+ if (tablets.isEmpty()) {
+ return null;
+ }
+
+ return new PreparedEntry(tablets, physicalTime, writerNodeId, localSeq, searchIndex);
+ }
+
+ private static long estimateTabletSize(final Tablet tablet) {
+ return PipeMemoryWeightUtil.calculateTabletSizeInBytes(tablet);
+ }
+
+ private void createAndEnqueueEvent(
+ final List<Tablet> tablets, final long startSearchIndex, final long endSearchIndex) {
+ createAndEnqueueEvent(
+ tablets, startSearchIndex, endSearchIndex, endSearchIndex, seekGeneration.get());
+ }
+
+ private boolean createAndEnqueueEvent(
+ final List<Tablet> tablets,
+ final long startSearchIndex,
+ final long endSearchIndex,
+ final long commitLocalSeq,
+ final long expectedSeekGeneration) {
+ if (tablets.isEmpty()) {
+ return true;
+ }
+
+ if (seekGeneration.get() != expectedSeekGeneration) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: skip stale event with searchIndex range [{}, {}], "
+ + "expectedSeekGeneration={}, currentSeekGeneration={}",
+ this,
+ startSearchIndex,
+ endSearchIndex,
+ expectedSeekGeneration,
+ seekGeneration.get());
+ return false;
+ }
+
+ final SubscriptionCommitContext commitContext = buildWriterCommitContext(commitLocalSeq);
+ final WriterId writerId = commitContext.getWriterId();
+ final WriterProgress writerProgress = commitContext.getWriterProgress();
+ commitManager.recordMapping(brokerId, topicName, consensusGroupId, writerId, writerProgress);
+
+ // nextOffset <= 0 means all tablets delivered in single batch
+ // -tablets.size() indicates total count
+ // Use Map<String, List<Tablet>> constructor with actual database name for table model;
+ final TabletsPayload payload =
+ new TabletsPayload(
+ Collections.singletonMap(converter.getDatabaseName(), tablets), -tablets.size());
+
+ final SubscriptionEvent event =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.TABLETS.getType(), payload, commitContext);
+
+ prefetchingQueue.add(event);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: ENQUEUED event with {} tablets, "
+ + "searchIndex range [{}, {}], prefetchQueueSize={}",
+ this,
+ tablets.size(),
+ startSearchIndex,
+ endSearchIndex,
+ prefetchingQueue.size());
+
+ // After enqueuing the data event, control metadata is handled separately from user data.
+ return true;
+ }
+
+ private SubscriptionCommitContext buildWriterCommitContext(final long localSeq) {
+ final int effectiveNodeId =
+ batchWriterNodeId >= 0
+ ? batchWriterNodeId
+ : IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), effectiveNodeId);
+ final WriterProgress writerProgress = new WriterProgress(batchPhysicalTime, localSeq);
+ return new SubscriptionCommitContext(
+ IoTDBDescriptor.getInstance().getConfig().getDataNodeId(),
+ PipeDataNodeAgent.runtime().getRebootTimes(),
+ topicName,
+ brokerId,
+ seekGeneration.get(),
+ writerId,
+ writerProgress);
+ }
+
+ private void updateBatchWriterProgress(final long physicalTime, final int writerNodeId) {
+ if (physicalTime > 0) {
+ this.batchPhysicalTime = physicalTime;
+ }
+ if (writerNodeId >= 0) {
+ this.batchWriterNodeId = writerNodeId;
+ }
+ }
+
+ private void resetBatchWriterProgress() {
+ this.batchPhysicalTime = 0L;
+ this.batchWriterNodeId = -1;
+ }
+
+ private long estimateTabletsBytes(final List<Tablet> tablets) {
+ long estimatedBytes = 0L;
+ for (final Tablet tablet : tablets) {
+ estimatedBytes += estimateTabletSize(tablet);
+ }
+ return estimatedBytes;
+ }
+
+ private boolean appendPreparedEntryViaRealtimeLane(
+ final DeliveryBatchState batchState,
+ final PreparedEntry preparedEntry,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ bufferRealtimeEntry(preparedEntry);
+ return drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes);
+ }
+
+ private int getRealtimeBufferedEntryCount() {
+ int count = 0;
+ for (final NavigableMap<Long, PreparedEntry> laneEntries : realtimeEntriesByLane.values()) {
+ count += laneEntries.size();
+ }
+ return count;
+ }
+
+ private boolean drainBufferedRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ while (!realtimeEntriesByLane.isEmpty()) {
+ final int bufferedBefore = getRealtimeBufferedEntryCount();
+ if (!drainRealtimeLanes(batchState, expectedSeekGeneration, maxTablets, maxBatchBytes)) {
+ return false;
+ }
+
+ final int bufferedAfter = getRealtimeBufferedEntryCount();
+ if (bufferedAfter == 0 || prefetchingQueue.size() >= MAX_PREFETCHING_QUEUE_SIZE) {
+ return true;
+ }
+
+ if (batchState.isEmpty()) {
+ return true;
+ }
+
+ if (!flushBatch(batchState, expectedSeekGeneration)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private boolean canAppendLaneEntry(
+ final DeliveryBatchState batchState,
+ final LaneBufferedEntry entry,
+ final long entryEstimatedBytes,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ final boolean wouldExceedEntryLimit =
+ maxEntries != Integer.MAX_VALUE && batchState.entryCount >= maxEntries;
+ final boolean wouldExceedTabletLimit =
+ !batchState.isEmpty() && batchState.tablets.size() + entry.getTablets().size() > maxTablets;
+ final boolean wouldExceedByteLimit =
+ !batchState.isEmpty() && batchState.estimatedBytes + entryEstimatedBytes > maxBatchBytes;
+ // Keep all consensus subscription modes on a single-writer commit/delivery shape so
+ // SubscriptionCommitContext and RegionProgress remain per-writer.
+ final boolean writerChanged =
+ !batchState.isEmpty() && batchState.writerNodeId != entry.getWriterNodeId();
+ return !(wouldExceedEntryLimit
+ || wouldExceedTabletLimit
+ || wouldExceedByteLimit
+ || writerChanged);
+ }
+
+ private boolean drainRealtimeLanes(
+ final DeliveryBatchState batchState,
+ final long expectedSeekGeneration,
+ final int maxTablets,
+ final long maxBatchBytes) {
+ return drainLaneEntries(
+ batchState,
+ this::buildRealtimeLaneFrontiers,
+ this::peekRealtimeEntry,
+ entry -> true,
+ (laneId, entry) -> removeRealtimeEntry(laneId, entry.localSeq),
+ Integer.MAX_VALUE,
+ maxTablets,
+ maxBatchBytes,
+ true);
+ }
+
+ private <T extends LaneBufferedEntry> boolean drainLaneEntries(
+ final DeliveryBatchState batchState,
+ final Supplier<PriorityQueue<LaneFrontier>> frontierSupplier,
+ final Function<WriterLaneId, T> headSupplier,
+ final Predicate<T> releasePredicate,
+ final BiConsumer<WriterLaneId, T> removeHeadAction,
+ final int maxEntries,
+ final int maxTablets,
+ final long maxBatchBytes,
+ final boolean trackLingerTime) {
+ while (true) {
+ final PriorityQueue<LaneFrontier> frontiers = frontierSupplier.get();
+ if (frontiers.isEmpty()) {
+ return true;
+ }
+ final LaneFrontier frontier = frontiers.peek();
+ if (Objects.isNull(frontier) || frontier.isBarrier) {
+ return true;
+ }
+ final T laneHead = headSupplier.apply(frontier.laneId);
+ if (Objects.isNull(laneHead)) {
+ return true;
+ }
+ if (!releasePredicate.test(laneHead)) {
+ return true;
+ }
+
+ final long entryEstimatedBytes = estimateTabletsBytes(laneHead.getTablets());
+ if (!canAppendLaneEntry(
+ batchState, laneHead, entryEstimatedBytes, maxEntries, maxTablets, maxBatchBytes)) {
+ return true;
+ }
+
+ removeHeadAction.accept(frontier.laneId, laneHead);
+ batchState.append(laneHead, entryEstimatedBytes, trackLingerTime);
+ }
+ }
+
+ private boolean flushBatch(
+ final DeliveryBatchState batchState, final long expectedSeekGeneration) {
+ updateBatchWriterProgress(batchState.physicalTime, batchState.writerNodeId);
+ if (!createAndEnqueueEvent(
+ new ArrayList<>(batchState.tablets),
+ batchState.startSearchIndex,
+ batchState.endSearchIndex,
+ batchState.lastLocalSeq,
+ expectedSeekGeneration)) {
+ return false;
+ }
+ resetBatchWriterProgress();
+ batchState.reset();
+ return true;
+ }
+
+ // ======================== Commit (Ack/Nack) ========================
+
+ private boolean canAcceptCommitContext(
+ final SubscriptionCommitContext commitContext, final String action, final boolean silent) {
+ if (isClosed || closeRequested || pendingSeekRequest != null) {
+ return false;
+ }
+ if (Objects.isNull(commitContext) || !commitContext.hasWriterProgress()) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} without writer progress, commitContext={}",
+ this,
+ action,
+ commitContext);
+ }
+ return false;
+ }
+ if (!isActive) {
+ if (silent) {
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ } else {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: reject {} for inactive queue, commitContext={}, runtimeVersion={}",
+ this,
+ action,
+ commitContext,
+ runtimeVersion);
+ }
+ return false;
+ }
+ return true;
+ }
+
+ public boolean ack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "ack", false)
+ && ackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean ackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ if (!acked.get()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for ack",
+ this,
+ commitContext);
+ }
+ return null;
+ }
+
+ if (ev.isCommitted()) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: event {} already committed", this, commitContext);
+ ev.cleanUp(false);
+ return null;
+ }
+
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to advance commit frontier for {}",
+ this,
+ commitContext);
+ return ev;
+ }
+
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+
+ return acked.get();
+ }
+
+ public boolean nack(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ return canAcceptCommitContext(commitContext, "nack", false)
+ && nackInternal(consumerId, commitContext);
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ /**
+ * Silent version of ack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean ackSilent(final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "ack", true)) {
+ return false;
+ }
+ final WriterId commitWriterId = extractCommitWriterId(commitContext);
+ final WriterProgress commitWriterProgress = extractCommitWriterProgress(commitContext);
+ final AtomicBoolean acked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ final boolean directCommitted =
+ commitManager.commitWithoutOutstanding(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ acked.set(directCommitted);
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ final boolean committed =
+ commitManager.commit(
+ brokerId, topicName, consensusGroupId, commitWriterId, commitWriterProgress);
+ if (!committed) {
+ return ev;
+ }
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ acked.set(true);
+ ev.cleanUp(false);
+ return null;
+ });
+ return acked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private WriterId extractCommitWriterId(final SubscriptionCommitContext commitContext) {
+ final WriterId writerId = commitContext.getWriterId();
+ return Objects.nonNull(writerId) ? writerId : new WriterId(consensusGroupId.toString(), -1);
+ }
+
+ private WriterProgress extractCommitWriterProgress(
+ final SubscriptionCommitContext commitContext) {
+ return commitContext.getWriterProgress();
+ }
+
+ /**
+ * Silent version of nack: returns false without logging if the commit context is not found. Used
+ * in multi-region iteration where only one queue owns the event.
+ */
+ public boolean nackSilent(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ acquireReadLock();
+ try {
+ if (!canAcceptCommitContext(commitContext, "nack", true)) {
+ return false;
+ }
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+ return nacked.get();
+ } finally {
+ releaseReadLock();
+ }
+ }
+
+ private boolean nackInternal(
+ final String consumerId, final SubscriptionCommitContext commitContext) {
+ final AtomicBoolean nacked = new AtomicBoolean(false);
+ inFlightEvents.compute(
+ new Pair<>(consumerId, commitContext),
+ (key, ev) -> {
+ if (Objects.isNull(ev)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: commit context {} does not exist for nack",
+ this,
+ commitContext);
+ return null;
+ }
+
+ ev.nack();
+ nacked.set(true);
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected (nackCount={}), "
+ + "force-acking event {} to prevent infinite re-delivery",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ return null;
+ });
+
+ return nacked.get();
+ }
+
+ // ======================== Recycle ========================
+
+ /** Recycles in-flight events that are pollable (timed out) back to the prefetching queue. */
+ private void recycleInFlightEvents() {
+ for (final Pair<String, SubscriptionCommitContext> key :
+ new ArrayList<>(inFlightEvents.keySet())) {
+ inFlightEvents.compute(
+ key,
+ (k, ev) -> {
+ if (Objects.isNull(ev)) {
+ return null;
+ }
+ if (ev.isCommitted()) {
+ ev.cleanUp(false);
+ return null;
+ }
+ if (ev.pollable()) {
+ ev.nack();
+ if (ev.isPoisoned()) {
+ LOGGER.error(
+ "ConsensusPrefetchingQueue {}: poison message detected during recycle "
+ + "(nackCount={}), force-acking event {}",
+ this,
+ ev.getNackCount(),
+ ev);
+ ev.ack();
+ ev.recordCommittedTimestamp();
+ ev.cleanUp(false);
+ return null;
+ }
+ prefetchingQueue.add(ev);
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: recycled timed-out event {} back to prefetching queue",
+ this,
+ ev);
+ return null;
+ }
+ return ev;
+ });
+ }
+ }
+
+ // ======================== Cleanup ========================
+
+ public void cleanUp() {
+ acquireWriteLock();
+ try {
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ pendingEntries.clear();
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+ pendingSubscriptionWalResetSearchIndex = Long.MIN_VALUE;
+ pendingSubscriptionWalResetGeneration = Long.MIN_VALUE;
+ closeSubscriptionWALIterator();
+
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ // ======================== Seek ========================
+
+ /**
+ * Seeks to the earliest available WAL position. The actual position depends on WAL retention: if
+ * old files have been reclaimed, the earliest available position may be later than 0.
+ */
+ public void seekToBeginning() {
+ seekToResolvedPosition(0L, new RegionProgress(Collections.emptyMap()), "beginning");
+ }
+
+ /**
+ * Seeks to the current WAL write position. After this, only newly written data will be consumed.
+ */
+ public void seekToEnd() {
+ seekToResolvedPosition(
+ consensusReqReader.getCurrentSearchIndex(), computeTailRegionProgress(), "end");
+ }
+
+ public void seekToRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress not supported (no WAL directory)",
+ this);
+ seekToBeginning();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, false);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekToRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgress");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekToRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ public void seekAfterRegionProgress(final RegionProgress regionProgress) {
+ if (!(consensusReqReader instanceof WALNode)) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress not supported (no WAL directory)",
+ this);
+ seekToEnd();
+ return;
+ }
+ final WALNode walNode = (WALNode) consensusReqReader;
+ walNode.rollWALFile();
+
+ final ReplayLocateDecision replayTarget =
+ locateReplayStartForRegionProgress(regionProgress, true);
+ switch (replayTarget.getStatus()) {
+ case FOUND:
+ case AT_END:
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seekAfterRegionProgress writerCount={} -> {} searchIndex={}",
+ this,
+ regionProgress.getWriterPositions().size(),
+ replayTarget.getStatus(),
+ replayTarget.getStartSearchIndex());
+ seekToResolvedPosition(
+ replayTarget.getStartSearchIndex(),
+ replayTarget.getRecoveryRegionProgress(),
+ "regionProgressAfter");
+ return;
+ case LOCATE_MISS:
+ default:
+ throw new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s: cannot seekAfterRegionProgress %s: %s",
+ this, regionProgress, replayTarget.getDetail()));
+ }
+ }
+
+ private synchronized void seekToResolvedPosition(
+ final long targetSearchIndex,
+ final RegionProgress committedRegionProgress,
+ final String seekReason) {
+ final PendingSeekRequest request;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ // Fence old commit contexts immediately. The grouped reset itself is applied later by the
+ // prefetch worker so WAL state and queue state still move under the queue's serial context.
+ final boolean previousPrefetchInitialized = prefetchInitialized;
+ final long previousSeekGeneration = seekGeneration.get();
+ final long targetSeekGeneration = seekGeneration.incrementAndGet();
+ request =
+ new PendingSeekRequest(
+ targetSearchIndex,
+ committedRegionProgress,
+ seekReason,
+ previousPrefetchInitialized,
+ previousSeekGeneration,
+ targetSeekGeneration);
+ pendingSeekRequest = request;
+ prefetchInitialized = true;
+ } finally {
+ releaseWriteLock();
+ }
+
+ final ConsensusPrefetchSubtask subtask = ensurePrefetchSubtaskBound();
+ if (Objects.isNull(subtask)) {
+ failPendingSeekBeforeScheduling(request);
+ request.awaitCompletion();
+ return;
+ }
+
+ subtask.requestWakeupNow();
+ request.awaitCompletion();
+ }
+
+ private boolean applyPendingSeekRequestIfNecessary() {
+ final PendingSeekRequest request = pendingSeekRequest;
+ if (Objects.isNull(request)) {
+ return false;
+ }
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return pendingSeekRequest != null;
+ }
+ pendingSeekRequest = null;
+ if (isClosed || closeRequested) {
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s is closing while applying seek", this)));
+ return true;
+ }
+ applySeekResetUnderWriteLock(request);
+ request.complete();
+ return true;
+ } catch (final RuntimeException e) {
+ request.fail(e);
+ throw e;
+ } finally {
+ releaseWriteLock();
+ }
+ }
+
+ public void abortPendingSeekForRuntimeStop() {
+ final PendingSeekRequest requestToFail;
+
+ acquireWriteLock();
+ try {
+ requestToFail = pendingSeekRequest;
+ if (Objects.isNull(requestToFail)) {
+ return;
+ }
+ pendingSeekRequest = null;
+ prefetchInitialized = requestToFail.previousPrefetchInitialized;
+ if (seekGeneration.get() == requestToFail.targetSeekGeneration) {
+ seekGeneration.set(requestToFail.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: aborted pending seek({}) during runtime stop, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ requestToFail.seekReason,
+ true,
+ requestToFail.previousPrefetchInitialized,
+ requestToFail.targetSeekGeneration,
+ requestToFail.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ requestToFail.fail(
+ new IllegalStateException(
+ String.format(
+ "ConsensusPrefetchingQueue %s runtime stopped before seek(%s) was applied",
+ this, requestToFail.seekReason)));
+ }
+
+ private void failPendingSeekBeforeScheduling(final PendingSeekRequest request) {
+ final boolean closing;
+
+ acquireWriteLock();
+ try {
+ if (pendingSeekRequest != request) {
+ return;
+ }
+ closing = isClosed || closeRequested;
+ pendingSeekRequest = null;
+ prefetchInitialized = request.previousPrefetchInitialized;
+ if (seekGeneration.get() == request.targetSeekGeneration) {
+ seekGeneration.set(request.previousSeekGeneration);
+ }
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: failed to schedule seek({}) because {}, restored prefetchInitialized {} -> {}, seekGeneration {} -> {}",
+ this,
+ request.seekReason,
+ closing ? "the queue is closing" : "prefetch runtime is unavailable",
+ true,
+ request.previousPrefetchInitialized,
+ request.targetSeekGeneration,
+ request.previousSeekGeneration);
+ } finally {
+ releaseWriteLock();
+ }
+
+ request.fail(
+ new IllegalStateException(
+ String.format(
+ closing
+ ? "ConsensusPrefetchingQueue %s is closing before seek(%s) can be scheduled"
+ : "ConsensusPrefetchingQueue %s cannot schedule seek(%s) because prefetch runtime is unavailable",
+ this,
+ request.seekReason)));
+ }
+
+ private void applySeekResetUnderWriteLock(final PendingSeekRequest request) {
+ // 1. Clean up all queued and in-flight events
+ prefetchingQueue.forEach(event -> event.cleanUp(true));
+ prefetchingQueue.clear();
+ inFlightEvents.values().forEach(event -> event.cleanUp(true));
+ inFlightEvents.clear();
+
+ // 2. Discard stale pending entries from in-memory queue
+ pendingEntries.clear();
+
+ // 3. Reset per-writer release state and source-level dedup frontiers.
+ realtimeEntriesByLane.clear();
+ writerLanes.clear();
+ clearRecoveryWriterProgress();
+ materializedFollowerProgressByWriter.clear();
+ if (Objects.nonNull(request.committedRegionProgress)
+ && !request.committedRegionProgress.getWriterPositions().isEmpty()) {
+ installRecoveryWriterProgress(request.committedRegionProgress);
+ }
+
+ // 4. Reset WAL read position
+ nextExpectedSearchIndex.set(request.targetSearchIndex);
+ requestSubscriptionWalReset(request.targetSearchIndex, seekGeneration.get());
+ lingerBatch.reset();
+ resetBatchWriterProgress();
+ observedSeekGeneration = seekGeneration.get();
+ pendingWalGapRetryRequested = false;
+ walGapWaitStartTimeMs = 0L;
+ lastWalGapWaitLogTimeMs = 0L;
+
+ // 5. Reset commit state to the writer progress immediately before the first re-delivered
+ // entry so seek/rebind resumes from the intended frontier.
+ commitManager.resetState(
+ brokerId, topicName, consensusGroupId, request.committedRegionProgress);
+
+ LOGGER.info(
+ "ConsensusPrefetchingQueue {}: seek({}) applied to searchIndex={}, writerCount={}, seekGeneration={}",
+ this,
+ request.seekReason,
+ request.targetSearchIndex,
+ Objects.nonNull(request.committedRegionProgress)
+ ? request.committedRegionProgress.getWriterPositions().size()
+ : 0,
+ seekGeneration.get());
+ }
+
+ private RegionProgress computeTailRegionProgress() {
+ if (!(consensusReqReader instanceof WALNode)) {
+ return new RegionProgress(Collections.emptyMap());
+ }
+
+ final WALNode walNode = (WALNode) consensusReqReader;
+ final Map<WriterId, WriterProgress> tailProgressByWriter = new LinkedHashMap<>();
+ final File[] walFiles = WALFileUtils.listAllWALFiles(walNode.getLogDirectory());
+ if (Objects.isNull(walFiles) || walFiles.length == 0) {
+ mergeTailProgress(tailProgressByWriter, walNode.getCurrentWALMetaDataSnapshot());
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ WALFileUtils.ascSortByVersionId(walFiles);
+ final long liveVersionId = walNode.getCurrentWALFileVersion();
+ final WALMetaData liveSnapshot = walNode.getCurrentWALMetaDataSnapshot();
+ for (final File walFile : walFiles) {
+ final long versionId = WALFileUtils.parseVersionId(walFile.getName());
+ if (versionId == liveVersionId) {
+ mergeTailProgress(tailProgressByWriter, liveSnapshot);
+ continue;
+ }
+ try (final ProgressWALReader reader = new ProgressWALReader(walFile)) {
+ mergeTailProgress(tailProgressByWriter, reader.getMetaData());
+ } catch (final IOException e) {
+ LOGGER.warn(
+ "ConsensusPrefetchingQueue {}: failed to read WAL metadata from {} while computing seekToEnd frontier",
+ this,
+ walFile,
+ e);
+ }
+ }
+ return new RegionProgress(tailProgressByWriter);
+ }
+
+ private void mergeTailProgress(
+ final Map<WriterId, WriterProgress> tailProgressByWriter, final WALMetaData metadata) {
+ if (Objects.isNull(metadata)) {
+ return;
+ }
+ final List<Long> physicalTimes = metadata.getPhysicalTimes();
+ final List<Short> nodeIds = metadata.getNodeIds();
+ final List<Long> localSeqs = metadata.getLocalSeqs();
+ final int size = Math.min(Math.min(physicalTimes.size(), nodeIds.size()), localSeqs.size());
+ for (int i = 0; i < size; i++) {
+ final int writerNodeId = nodeIds.get(i);
+ final long physicalTime = physicalTimes.get(i);
+ final long localSeq = localSeqs.get(i);
+ if (writerNodeId < 0 || physicalTime < 0L || localSeq < 0L) {
+ continue;
+ }
+
+ final WriterId writerId = new WriterId(consensusGroupId.toString(), writerNodeId);
+ final WriterProgress candidateProgress = new WriterProgress(physicalTime, localSeq);
+ final WriterProgress currentProgress = tailProgressByWriter.get(writerId);
+ if (Objects.isNull(currentProgress)
+ || compareWriterProgress(candidateProgress, currentProgress) > 0) {
+ tailProgressByWriter.put(writerId, candidateProgress);
+ }
+ }
+ }
+
+ /**
+ * Extracts the maximum timestamp from an InsertNode. For row nodes this is the single timestamp;
+ * for tablet nodes, {@code times} is sorted so the last element is the max. For composite nodes,
+ * iterates over children.
+ *
+ * @return the maximum timestamp, or {@code Long.MIN_VALUE} if extraction fails
+ */
+ private long extractMaxTime(final InsertNode insertNode) {
+ try {
+ if (insertNode instanceof InsertRowNode) {
+ return ((InsertRowNode) insertNode).getTime();
+ }
+ if (insertNode instanceof InsertTabletNode) {
+ final InsertTabletNode tabletNode = (InsertTabletNode) insertNode;
+ final int rowCount = tabletNode.getRowCount();
+ return rowCount > 0 ? tabletNode.getTimes()[rowCount - 1] : Long.MIN_VALUE;
+ }
+ if (insertNode instanceof InsertMultiTabletsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertTabletNode child :
+ ((InsertMultiTabletsNode) insertNode).getInsertTabletNodeList()) {
+ final int rowCount = child.getRowCount();
+ if (rowCount > 0) {
+ max = Math.max(max, child.getTimes()[rowCount - 1]);
+ }
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row : ((InsertRowsNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ if (insertNode instanceof InsertRowsOfOneDeviceNode) {
+ long max = Long.MIN_VALUE;
+ for (final InsertRowNode row :
+ ((InsertRowsOfOneDeviceNode) insertNode).getInsertRowNodeList()) {
+ max = Math.max(max, row.getTime());
+ }
+ return max;
+ }
+ // Fallback: use getMinTime() which at least gets a timestamp
+ return insertNode.getMinTime();
+ } catch (final Exception e) {
+ return Long.MIN_VALUE;
+ }
+ }
+
+ /**
+ * Checks whether it is time to inject a watermark event and does so if the configured interval
+ * has elapsed. Called from prefetch rounds after processing data and during idle scheduling.
+ */
+ private void maybeInjectWatermark() {
+ if (maxObservedTimestamp == Long.MIN_VALUE) {
+ return; // No data observed yet, nothing to report
+ }
+ final long intervalMs =
+ SubscriptionConfig.getInstance().getSubscriptionConsensusWatermarkIntervalMs();
+ if (intervalMs <= 0) {
+ return; // Watermark disabled
+ }
+ final long now = System.currentTimeMillis();
+ if (now - lastWatermarkEmitTimeMs >= intervalMs) {
+ injectWatermark(maxObservedTimestamp);
+ lastWatermarkEmitTimeMs = now;
+ }
+ }
+
+ /**
+ * Injects a {@link SubscriptionPollResponseType#WATERMARK} event into the prefetching queue. The
+ * committed mapping is deliberately NOT recorded because watermark events are metadata, not user
+ * data.
+ *
+ * @param watermarkTimestamp the maximum data timestamp observed so far
+ */
+ private void injectWatermark(final long watermarkTimestamp) {
+ final int dataNodeId = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ final SubscriptionCommitContext watermarkCtx = createNonCommittableSeekContext(dataNodeId);
+ final SubscriptionEvent watermarkEvent =
+ new SubscriptionEvent(
+ SubscriptionPollResponseType.WATERMARK.getType(),
+ new WatermarkPayload(watermarkTimestamp, dataNodeId),
+ watermarkCtx);
+ prefetchingQueue.add(watermarkEvent);
+
+ LOGGER.debug(
+ "ConsensusPrefetchingQueue {}: injected WATERMARK, watermarkTimestamp={}",
+ this,
+ watermarkTimestamp);
+ }
+
+ /** Returns the maximum observed data timestamp for metrics. */
+ public long getMaxObservedTimestamp() {
+ return maxObservedTimestamp;
+ }
+
+ private void markAcceptedFromPending() {
+ pendingPathAcceptedEntries.incrementAndGet();
+ }
+
+ private void markAcceptedFromWal() {
+ walPathAcceptedEntries.incrementAndGet();
+ }
+
+ public void close() {
+ final PendingSeekRequest seekRequestToFail;
+ final Pair<ConsensusSubscriptionPrefetchExecutor, ConsensusPrefetchSubtask> prefetchBinding;
+
+ acquireWriteLock();
+ try {
+ if (isClosed || closeRequested) {
+ return;
+ }
+ closeRequested = true;
+ seekRequestToFail = pendingSeekRequest;
+ pendingSeekRequest = null;
+ } finally {
+ releaseWriteLock();
+ }
+
+ prefetchBinding = detachPrefetchSubtask();
+
+ if (Objects.nonNull(seekRequestToFail)) {
+ seekRequestToFail.fail(
+ new IllegalStateException(
+ String.format("ConsensusPrefetchingQueue %s is closing before seek applies", this)));
+ }
+
+ if (Objects.nonNull(prefetchBinding.right)) {
+ prefetchBinding.right.cancelPendingExecution();
+ prefetchBinding.right.awaitIdle();
+ }
+
+ try {
+ acquireWriteLock();
+ try {
+ if (!isClosed
+ && pendingSeekRequest == null
+ && seekGeneration.get() == observedSeekGeneration) {
+ flushLingeringBatchOnCloseUnderWriteLock();
+ }
+ markClosed();
+ } finally {
+ releaseWriteLock();
+ }
+
+ // Deregister metrics after the queue is fully closed.
+ ConsensusSubscriptionPrefetchingQueueMetrics.getInstance()
+ .deregister(getPrefetchingQueueId());
+
+ if (Objects.nonNull(prefetchBinding.left) && Objects.nonNull(prefetchBinding.right)) {
+ if (!prefetchBinding.left.isShutdown()) {
+ prefetchBinding.left.deregister(prefetchBinding.right.getTaskId());
+ } else {
+ prefetchBinding.right.close();
+ }
+ }
+
+ try {
+ // Deregister from IoTConsensusServerImpl (stop receiving in-memory data).
+ serverImpl.deregisterSubscriptionQueue(pendingEntries);
+ } catch (final Exception e) {
+ LOGGER.warn("ConsensusPrefetchingQueue {}: error during deregister", this, e);
+ } finally {
+ try {
+ cleanUp();
+ } finally {
+ // Persist progress before closing
+ commitManager.persistAll();
+ }
+ }
Review Comment:
Adjusted close() to deregister the pending subscription queue before flushing lingering batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3279146543
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/consensus/ConsensusLogToTabletConverter.java:
##########
@@ -0,0 +1,606 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.subscription.broker.consensus;
+
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TablePattern;
+import org.apache.iotdb.commons.pipe.datastructure.pattern.TreePattern;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertMultiTabletsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertRowsOfOneDeviceNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.InsertTabletNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertRowsNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.write.RelationalInsertTabletNode;
+
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+import java.util.regex.Pattern;
+
+/** Converts IoTConsensus WAL log entries (InsertNode) to Tablet format for subscription. */
+public class ConsensusLogToTabletConverter {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(ConsensusLogToTabletConverter.class);
+
+ private final TreePattern treePattern;
+ private final TablePattern tablePattern;
+ private final Pattern tableColumnPattern;
+
+ /**
+ * The actual database name of the DataRegion this converter processes (table-model format without
+ * "root." prefix). Null for tree-model topics.
+ */
+ private final String databaseName;
+
+ public ConsensusLogToTabletConverter(
+ final TreePattern treePattern,
+ final TablePattern tablePattern,
+ final Pattern tableColumnPattern,
+ final String databaseName) {
+ this.treePattern = treePattern;
+ this.tablePattern = tablePattern;
+ this.tableColumnPattern = tableColumnPattern;
+ this.databaseName = databaseName;
+ }
+
+ public String getDatabaseName() {
+ return databaseName;
+ }
+
+ static String safeDeviceIdForLog(final InsertNode node) {
+ try {
+ final Object deviceId = node.getDeviceID();
+ return deviceId != null ? deviceId.toString() : "null";
+ } catch (final Exception e) {
+ return "N/A(" + node.getType() + ")";
+ }
+ }
+
+ public List<Tablet> convert(final InsertNode insertNode) {
+ if (Objects.isNull(insertNode)) {
+ return Collections.emptyList();
+ }
+
+ final PlanNodeType nodeType = insertNode.getType();
+ if (nodeType == null) {
+ LOGGER.warn("InsertNode type is null, skipping conversion");
+ return Collections.emptyList();
+ }
+
+ LOGGER.debug(
+ "ConsensusLogToTabletConverter: converting InsertNode type={}, deviceId={}",
+ nodeType,
+ safeDeviceIdForLog(insertNode));
+
+ switch (nodeType) {
+ case INSERT_ROW:
+ return convertInsertRowNode((InsertRowNode) insertNode);
+ case INSERT_TABLET:
+ return convertInsertTabletNode((InsertTabletNode) insertNode);
+ case INSERT_ROWS:
+ return convertInsertRowsNode((InsertRowsNode) insertNode);
+ case INSERT_ROWS_OF_ONE_DEVICE:
+ return convertInsertRowsOfOneDeviceNode((InsertRowsOfOneDeviceNode) insertNode);
+ case INSERT_MULTI_TABLET:
+ return convertInsertMultiTabletsNode((InsertMultiTabletsNode) insertNode);
+ case RELATIONAL_INSERT_ROW:
+ return convertRelationalInsertRowNode((RelationalInsertRowNode) insertNode);
+ case RELATIONAL_INSERT_TABLET:
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) insertNode);
+ case RELATIONAL_INSERT_ROWS:
+ return convertRelationalInsertRowsNode((RelationalInsertRowsNode) insertNode);
+ default:
+ LOGGER.debug("Unsupported InsertNode type for subscription: {}", nodeType);
+ return Collections.emptyList();
+ }
+ }
+
+ // ======================== Tree Model Conversion ========================
+
+ private List<Tablet> convertInsertRowNode(final InsertRowNode node) {
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final long time = node.getTime();
+
+ // Determine which columns match the pattern
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final Object[] values = node.getValues();
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // Build Tablet with matched columns
+ final int columnCount = matchedColumnIndices.size();
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ final Tablet tablet = new Tablet(deviceId.toString(), schemas, 1 /* maxRowNumber */);
+ tablet.addTimestamp(0, time);
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = matchedColumnIndices.get(i);
+ final Object value = values[originalColIdx];
+ if (value == null) {
+ if (tablet.getBitMaps() == null) {
+ tablet.initBitMaps();
+ }
+ tablet.getBitMaps()[i].mark(0);
+ } else {
+ addValueToTablet(tablet, 0, i, dataTypes[originalColIdx], value);
+ }
+ }
+ tablet.setRowSize(1);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertTabletNode(final InsertTabletNode node) {
+ if (node instanceof RelationalInsertTabletNode) {
+ return convertRelationalInsertTabletNode((RelationalInsertTabletNode) node);
+ }
+
+ final IDeviceID deviceId = node.getDeviceID();
+
+ // Device-level path filtering
+ if (treePattern != null && !treePattern.mayOverlapWithDevice(deviceId)) {
+ return Collections.emptyList();
+ }
+
+ final String[] measurements = node.getMeasurements();
+ final TSDataType[] dataTypes = node.getDataTypes();
+ final long[] times = node.getTimes();
+ final Object[] columns = node.getColumns();
+ final BitMap[] bitMaps = node.getBitMaps();
+ final int rowCount = node.getRowCount();
+
+ // Column filtering
+ final List<Integer> matchedColumnIndices = getMatchedTreeColumnIndices(deviceId, measurements);
+ if (matchedColumnIndices.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ final int columnCount = matchedColumnIndices.size();
+ final boolean allColumnsMatch = (columnCount == measurements.length);
+
+ // Build schemas (always needed)
+ final List<IMeasurementSchema> schemas = new ArrayList<>(columnCount);
+ for (final int colIdx : matchedColumnIndices) {
+ schemas.add(new MeasurementSchema(measurements[colIdx], dataTypes[colIdx]));
+ }
+
+ // Build column arrays and bitmaps using bulk copy
+ final long[] newTimes = Arrays.copyOf(times, rowCount);
+ final Object[] newColumns = new Object[columnCount];
+ final BitMap[] newBitMaps = new BitMap[columnCount];
+
+ for (int i = 0; i < columnCount; i++) {
+ final int originalColIdx = allColumnsMatch ? i : matchedColumnIndices.get(i);
+ newColumns[i] = copyColumnArray(dataTypes[originalColIdx], columns[originalColIdx], rowCount);
+ if (bitMaps != null && bitMaps[originalColIdx] != null) {
+ newBitMaps[i] = new BitMap(rowCount);
+ BitMap.copyOfRange(bitMaps[originalColIdx], 0, newBitMaps[i], 0, rowCount);
+ }
+ }
+
+ final Tablet tablet =
+ new Tablet(deviceId.toString(), schemas, newTimes, newColumns, newBitMaps, rowCount);
+
+ return Collections.singletonList(tablet);
+ }
+
+ private List<Tablet> convertInsertRowsNode(final InsertRowsNode node) {
+ final List<Tablet> tablets = new ArrayList<>();
+ for (final InsertRowNode rowNode : node.getInsertRowNodeList()) {
+ // Handle merge bug: RelationalInsertRowNode.mergeInsertNode() is not overridden,
+ // so merged relational nodes arrive as InsertRowsNode (tree) with RelationalInsertRowNode
+ // children. Dispatch correctly by checking the actual child type.
+ if (rowNode instanceof RelationalInsertRowNode) {
+ tablets.addAll(convertRelationalInsertRowNode((RelationalInsertRowNode) rowNode));
+ } else {
+ tablets.addAll(convertInsertRowNode(rowNode));
+ }
+ }
+ return tablets;
+ }
+
+ private List<Tablet> convertInsertRowsOfOneDeviceNode(final InsertRowsOfOneDeviceNode node) {
+ final List<Tablet> tablets = new ArrayList<>();
+ for (final InsertRowNode rowNode : node.getInsertRowNodeList()) {
+ tablets.addAll(convertInsertRowNode(rowNode));
+ }
+ return tablets;
+ }
Review Comment:
Grouped adjacent tree-model rows with the same device/schema into one Tablet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "DanielWang2035 (via GitHub)" <gi...@apache.org>.
DanielWang2035 commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3278680701
##########
iotdb-client/subscription/src/main/java/org/apache/iotdb/session/subscription/consumer/base/AbstractSubscriptionConsumer.java:
##########
@@ -1390,6 +1628,365 @@ private void unsubscribeWithRedirection(final Set<String> topicNames)
throw new SubscriptionRuntimeCriticalException(errorMessage);
}
+ /**
+ * Sends seek request to ALL available providers. Unlike subscribe/unsubscribe, seek is only
+ * considered successful if every available provider acknowledges it because data regions for the
+ * topic may be distributed across different nodes.
+ */
+ private void seekWithRedirection(
+ final String topicName, final short seekType, final long timestamp)
+ throws SubscriptionException {
+ final List<AbstractSubscriptionProvider> providers = this.providers.getAllAvailableProviders();
+ if (providers.isEmpty()) {
+ throw new SubscriptionConnectionException(
+ String.format(
+ "Cluster has no available subscription providers when %s seek topic %s",
+ this, topicName));
+ }
+ final List<AbstractSubscriptionProvider> failedProviders = new ArrayList<>();
+ Throwable firstFailure = null;
+ for (final AbstractSubscriptionProvider provider : providers) {
+ try {
+ provider.seek(topicName, seekType, timestamp);
+ } catch (final Exception e) {
+ failedProviders.add(provider);
+ if (Objects.isNull(firstFailure)) {
+ firstFailure = e;
+ }
+ LOGGER.warn(
+ "{} failed to seek topic {} from subscription provider {}; seek requires every provider to succeed, so the client will continue notifying the remaining providers before failing this seek.",
+ this,
+ topicName,
+ provider,
+ e);
+ }
+ }
+ if (!failedProviders.isEmpty()) {
+ final String errorMessage =
+ String.format(
+ "%s failed to seek topic %s on subscription providers %s; seek requires every available provider to succeed",
+ this, topicName, failedProviders);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionRuntimeCriticalException(errorMessage, firstFailure);
+ }
+ }
+
+ /** Same all-provider success requirement as {@link #seekWithRedirection(String, short, long)}. */
+ private void seekWithRedirectionTopicProgress(
+ final String topicName, final TopicProgress topicProgress) throws SubscriptionException {
+ final List<AbstractSubscriptionProvider> providers = this.providers.getAllAvailableProviders();
+ if (providers.isEmpty()) {
+ throw new SubscriptionConnectionException(
+ String.format(
+ "Cluster has no available subscription providers when %s seek topic %s",
+ this, topicName));
+ }
+ final List<AbstractSubscriptionProvider> failedProviders = new ArrayList<>();
+ Throwable firstFailure = null;
+ for (final AbstractSubscriptionProvider provider : providers) {
+ try {
+ provider.seekToTopicProgress(topicName, topicProgress);
+ } catch (final Exception e) {
+ failedProviders.add(provider);
+ if (Objects.isNull(firstFailure)) {
+ firstFailure = e;
+ }
+ LOGGER.warn(
+ "{} failed to seek topic {} to topicProgress(regionCount={}) from provider {}; seek requires every provider to succeed, so the client will continue notifying the remaining providers before failing this seek.",
+ this,
+ topicName,
+ topicProgress.getRegionProgress().size(),
+ provider,
+ e);
+ }
+ }
+ if (!failedProviders.isEmpty()) {
+ final String errorMessage =
+ String.format(
+ "%s failed to seek topic %s to topicProgress(regionCount=%d) on subscription providers %s; seek requires every available provider to succeed",
+ this, topicName, topicProgress.getRegionProgress().size(), failedProviders);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionRuntimeCriticalException(errorMessage, firstFailure);
+ }
+ }
+
+ /** Same all-provider success requirement as {@link #seekWithRedirection(String, short, long)}. */
+ private void seekAfterWithRedirectionTopicProgress(
+ final String topicName, final TopicProgress topicProgress) throws SubscriptionException {
+ final List<AbstractSubscriptionProvider> providers = this.providers.getAllAvailableProviders();
+ if (providers.isEmpty()) {
+ throw new SubscriptionConnectionException(
+ String.format(
+ "Cluster has no available subscription providers when %s seekAfter topic %s",
+ this, topicName));
+ }
+ final List<AbstractSubscriptionProvider> failedProviders = new ArrayList<>();
+ Throwable firstFailure = null;
+ for (final AbstractSubscriptionProvider provider : providers) {
+ try {
+ provider.seekAfterTopicProgress(topicName, topicProgress);
+ } catch (final Exception e) {
+ failedProviders.add(provider);
+ if (Objects.isNull(firstFailure)) {
+ firstFailure = e;
+ }
+ LOGGER.warn(
+ "{} failed to seekAfter topic {} to topicProgress(regionCount={}) from provider {}; seek requires every provider to succeed, so the client will continue notifying the remaining providers before failing this seekAfter.",
+ this,
+ topicName,
+ topicProgress.getRegionProgress().size(),
+ provider,
+ e);
+ }
+ }
+ if (!failedProviders.isEmpty()) {
+ final String errorMessage =
+ String.format(
+ "%s failed to seekAfter topic %s to topicProgress(regionCount=%d) on subscription providers %s; seek requires every available provider to succeed",
+ this, topicName, topicProgress.getRegionProgress().size(), failedProviders);
+ LOGGER.warn(errorMessage);
+ throw new SubscriptionRuntimeCriticalException(errorMessage, firstFailure);
+ }
+ }
+
+ private Map<String, TopicProgress> buildCurrentProgressByTopic(final Set<String> topicNames) {
+ final Map<String, TopicProgress> result = new HashMap<>();
+ for (final String topicName : topicNames) {
+ final TopicProgress topicProgress = currentPositionsByTopic.get(topicName);
+ if (Objects.isNull(topicProgress) || topicProgress.getRegionProgress().isEmpty()) {
+ continue;
+ }
+ result.put(topicName, new TopicProgress(topicProgress.getRegionProgress()));
+ }
+ return result;
+ }
+
+ private void advanceCurrentPositions(final List<SubscriptionMessage> messages) {
+ for (final SubscriptionMessage message : messages) {
+ final SubscriptionCommitContext commitContext = message.getCommitContext();
+ if (Objects.isNull(commitContext) || Objects.isNull(commitContext.getTopicName())) {
+ continue;
+ }
+ mergeTopicProgress(
+ currentPositionsByTopic,
+ commitContext.getTopicName(),
+ extractWriterId(commitContext),
+ extractWriterProgress(commitContext));
+ }
+ }
+
+ private void advanceCommittedPositions(
+ final List<SubscriptionCommitContext> subscriptionCommitContexts) {
+ for (final SubscriptionCommitContext commitContext : subscriptionCommitContexts) {
+ if (Objects.isNull(commitContext) || Objects.isNull(commitContext.getTopicName())) {
+ continue;
+ }
+ mergeTopicProgress(
+ committedPositionsByTopic,
+ commitContext.getTopicName(),
+ extractWriterId(commitContext),
+ extractWriterProgress(commitContext));
+ }
+ }
+
+ private boolean isConsensusCommitContext(final SubscriptionCommitContext commitContext) {
+ return Objects.nonNull(commitContext)
+ && Objects.nonNull(commitContext.getWriterId())
+ && Objects.nonNull(commitContext.getWriterProgress())
+ && Objects.nonNull(commitContext.getRegionId())
+ && !commitContext.getRegionId().isEmpty();
+ }
+
+ private String buildTopicRegionKey(final SubscriptionCommitContext commitContext) {
+ return commitContext.getTopicName() + '\u0001' + commitContext.getRegionId();
+ }
+
+ private void stagePendingRedirectAck(final SubscriptionCommitContext commitContext) {
+ pendingRedirectAcksByTopicRegion
+ .computeIfAbsent(
+ buildTopicRegionKey(commitContext), ignored -> ConcurrentHashMap.newKeySet())
+ .add(commitContext);
+ }
+
+ private void flushPendingRedirectAcks(final List<SubscriptionMessage> currentMessages) {
+ final Map<String, Integer> redirectTargetByTopicRegion = new HashMap<>();
+ for (final SubscriptionMessage message : currentMessages) {
+ final SubscriptionCommitContext commitContext = message.getCommitContext();
+ if (!isConsensusCommitContext(commitContext)) {
+ continue;
+ }
+ redirectTargetByTopicRegion.put(
+ buildTopicRegionKey(commitContext), commitContext.getDataNodeId());
+ }
+
+ for (final Entry<String, Integer> entry : redirectTargetByTopicRegion.entrySet()) {
+ final Set<SubscriptionCommitContext> pendingContexts =
+ pendingRedirectAcksByTopicRegion.get(entry.getKey());
+ if (Objects.isNull(pendingContexts) || pendingContexts.isEmpty()) {
+ continue;
+ }
+
+ final List<SubscriptionCommitContext> contextsToRedirect = new ArrayList<>(pendingContexts);
+ try {
+ commitInternal(entry.getValue(), contextsToRedirect, false);
+ advanceCommittedPositions(contextsToRedirect);
+ contextsToRedirect.forEach(pendingContexts::remove);
+ if (pendingContexts.isEmpty()) {
+ pendingRedirectAcksByTopicRegion.remove(entry.getKey(), pendingContexts);
+ }
+ } catch (final SubscriptionException e) {
+ LOGGER.warn(
+ "{} failed to redirect {} pending consensus ack(s) for {} via provider {}",
+ this,
+ contextsToRedirect.size(),
+ entry.getKey(),
+ entry.getValue(),
+ e);
+ }
+ }
+ }
+
+ private boolean isNewerWriterProgress(
+ final long newPhysicalTime,
+ final long newLocalSeq,
+ final long oldPhysicalTime,
+ final long oldLocalSeq) {
+ return newPhysicalTime > oldPhysicalTime
+ || (newPhysicalTime == oldPhysicalTime && newLocalSeq > oldLocalSeq);
+ }
+
+ private void clearCurrentPositions(final String topicName) {
+ currentPositionsByTopic.remove(topicName);
+ }
+
+ private void clearCommittedPositions(final String topicName) {
+ committedPositionsByTopic.remove(topicName);
+ }
+
+ private void clearPendingRedirectAcks(final String topicName) {
+ final String prefix = topicName + '\u0001';
+ pendingRedirectAcksByTopicRegion.keySet().removeIf(key -> key.startsWith(prefix));
+ }
+
+ private void setCurrentPositions(final String topicName, final TopicProgress topicProgress) {
+ if (Objects.isNull(topicProgress) || topicProgress.getRegionProgress().isEmpty()) {
+ currentPositionsByTopic.remove(topicName);
+ return;
+ }
+ currentPositionsByTopic.put(topicName, new TopicProgress(topicProgress.getRegionProgress()));
+ }
+
+ private void setCommittedPositions(final String topicName, final TopicProgress topicProgress) {
+ if (Objects.isNull(topicProgress) || topicProgress.getRegionProgress().isEmpty()) {
+ committedPositionsByTopic.remove(topicName);
+ return;
+ }
+ committedPositionsByTopic.put(topicName, new TopicProgress(topicProgress.getRegionProgress()));
+ }
+
+ private void overlayCurrentPositions(final String topicName, final TopicProgress topicProgress) {
+ overlayTopicProgress(currentPositionsByTopic, topicName, topicProgress);
+ }
+
+ private void overlayCommittedPositions(
+ final String topicName, final TopicProgress topicProgress) {
+ overlayTopicProgress(committedPositionsByTopic, topicName, topicProgress);
+ }
+
+ private void overlayTopicProgress(
+ final Map<String, TopicProgress> progressByTopic,
+ final String topicName,
+ final TopicProgress topicProgress) {
+ if (Objects.isNull(topicName)
+ || topicName.isEmpty()
+ || Objects.isNull(topicProgress)
+ || topicProgress.getRegionProgress().isEmpty()) {
+ return;
+ }
+ progressByTopic.compute(
+ topicName,
+ (ignored, oldTopicProgress) -> {
+ final Map<String, RegionProgress> mergedRegionProgress =
+ Objects.nonNull(oldTopicProgress)
+ ? new HashMap<>(oldTopicProgress.getRegionProgress())
+ : new HashMap<>();
+ topicProgress
+ .getRegionProgress()
+ .forEach(
+ (regionId, regionProgress) -> {
+ if (Objects.isNull(regionId)
+ || regionId.isEmpty()
+ || Objects.isNull(regionProgress)
+ || regionProgress.getWriterPositions().isEmpty()) {
+ return;
+ }
+ mergedRegionProgress.put(
+ regionId,
+ new RegionProgress(new HashMap<>(regionProgress.getWriterPositions())));
+ });
Review Comment:
To avoid mutating a progress object that may still be referenced elsewhere. The merged result is treated as a new snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Subscription: implement IoTConsensus-based subscription [iotdb]
Posted by "VGalaxies (via GitHub)" <gi...@apache.org>.
VGalaxies commented on code in PR #17238:
URL: https://github.com/apache/iotdb/pull/17238#discussion_r3346560590
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/wal/utils/WALFileUtils.java:
##########
@@ -182,4 +197,243 @@ public static String getTsFileRelativePath(String absolutePath) {
Path path = new File(absolutePath).toPath();
return path.subpath(path.getNameCount() - 5, path.getNameCount()).toString();
}
+
+ /**
+ * Locate the first local searchIndex whose writer progress is equal to or strictly greater than
+ * the given writer-local frontier. This is currently used by single-writer recovery paths, so it
+ * matches only entries from the supplied nodeId.
+ *
+ * @return [targetSearchIndex, exactMatchFlag], or null if no matching/later entry exists
+ */
+ public static long[] locateByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] exactSearchIndex = new long[] {-1L};
+ final long[] firstAfterSearchIndex = new long[] {-1L};
+ final long[] firstAfterPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] firstAfterLocalSeq = new long[] {Long.MAX_VALUE};
+
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ final int cmp =
+ compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq);
+ if (cmp == 0) {
+ exactSearchIndex[0] = request.searchIndex;
+ return false;
+ }
+ if (cmp > 0
+ && (firstAfterSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ firstAfterPhysicalTime[0],
+ nodeId,
+ firstAfterLocalSeq[0])
+ < 0)) {
+ firstAfterSearchIndex[0] = request.searchIndex;
+ firstAfterPhysicalTime[0] = request.physicalTime;
+ firstAfterLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+
+ if (exactSearchIndex[0] >= 0L) {
+ return new long[] {exactSearchIndex[0], 1L};
+ }
+ if (firstAfterSearchIndex[0] >= 0L) {
+ return new long[] {firstAfterSearchIndex[0], 0L};
+ }
+ return null;
+ }
+
+ public static long findSearchIndexByWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] located = locateByWriterProgress(logDir, nodeId, physicalTime, localSeq);
+ return located != null && located[1] == 1L ? located[0] : -1L;
+ }
+
+ public static long findSearchIndexAfterWriterProgress(
+ final File logDir, final int nodeId, final long physicalTime, final long localSeq) {
+ final long[] bestSearchIndex = new long[] {-1L};
+ final long[] bestPhysicalTime = new long[] {Long.MAX_VALUE};
+ final long[] bestLocalSeq = new long[] {Long.MAX_VALUE};
+ forEachSealedSearchableRequest(
+ logDir,
+ request -> {
+ if (request.nodeId != nodeId) {
+ return true;
+ }
+ if (compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ physicalTime,
+ nodeId,
+ localSeq)
+ <= 0) {
+ return true;
+ }
+ if (bestSearchIndex[0] < 0L
+ || compareWriterProgress(
+ request.physicalTime,
+ request.nodeId,
+ request.localSeq,
+ bestPhysicalTime[0],
+ nodeId,
+ bestLocalSeq[0])
+ < 0) {
+ bestSearchIndex[0] = request.searchIndex;
+ bestPhysicalTime[0] = request.physicalTime;
+ bestLocalSeq[0] = request.localSeq;
+ }
+ return true;
+ });
+ return bestSearchIndex[0];
+ }
+
+ private interface SearchableRequestVisitor {
+ boolean onRequest(SearchableRequestMeta request);
Review Comment:
Done in 5ea428025e. Added a Javadoc note explaining that the method returns the first search index strictly greater than the writer progress, or -1 when none exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org