You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "zerolbsony (via GitHub)" <gi...@apache.org> on 2026/03/09 12:48:52 UTC
[PR] Automatically repair lost data partitions [iotdb]
zerolbsony opened a new pull request, #17279:
URL: https://github.com/apache/iotdb/pull/17279
After the ConfigNode restarts, it will automatically start checking whether the data partition tables are complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944159795
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -147,6 +160,11 @@ protected void start() throws IoTDBException {
}
active();
LOGGER.info("IoTDB started");
+ try {
+ dataPartitionTableCheckFuture.get();
+ } catch (ExecutionException | InterruptedException e) {
+ LOGGER.error("Data partition table check task execute failed", e);
Review Comment:
Changed yet
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/enums/DataPartitionTableGeneratorState.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.commons.enums;
+
+public enum DataPartitionTableGeneratorState {
+ SUCCESS(0),
+ FAILED(1),
+ IN_PROGRESS(2),
+ UNKNOWN(-1);
+
+ private final int code;
+
+ DataPartitionTableGeneratorState(int code) {
+ this.code = code;
+ }
+
+ public int getCode() {
+ return code;
+ }
+
+ /**
+ * get DataNodeRemoveState by code
+ *
+ * @param code code
+ * @return DataNodeRemoveState
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944067177
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
Review Comment:
The executor is no longer a global parameter of class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "CRZbulabula (via GitHub)" <gi...@apache.org>.
CRZbulabula commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944736652
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
Review Comment:
For all log with level > DEBUG, you'd better append '[DataPartitionIntegrity]' prefix, easier for your debug.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3138,349 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+ // Set default value
+ resp.setDatabaseScopedDataPartitionTables(Collections.emptyList());
+ try {
+ currentGeneratorFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ parseGenerationStatus(resp);
+ if (currentGenerator.getStatus().equals(DataPartitionTableGenerator.TaskStatus.COMPLETED)) {
+ boolean success = false;
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = new ArrayList<>();
+ Map<String, DataPartitionTable> dataPartitionTableMap = currentGenerator.getDatabasePartitionTableMap();
+ if (!dataPartitionTableMap.isEmpty()) {
+ for (Map.Entry<String, DataPartitionTable> entry : dataPartitionTableMap.entrySet()) {
+ String database = entry.getKey();
+ DataPartitionTable dataPartitionTable = entry.getValue();
+ if (!StringUtils.isEmpty(database) && dataPartitionTable != null) {
+ DatabaseScopedDataPartitionTable databaseScopedDataPartitionTable = new DatabaseScopedDataPartitionTable(database, dataPartitionTable);
+ databaseScopedDataPartitionTableList.add(databaseScopedDataPartitionTable);
+ success = true;
+ }
+ }
+ }
+
+ if (success) {
+ List<ByteBuffer> result = serializeDatabaseScopedTableList(databaseScopedDataPartitionTableList);
+ resp.setDatabaseScopedDataPartitionTables(result);
+
+ // Clear current generator
+ currentGenerator = null;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+ return resp;
+ }
+
+ private <T> void parseGenerationStatus(T resp) {
+ if (resp instanceof TGenerateDataPartitionTableResp) {
+ handleResponse((TGenerateDataPartitionTableResp) resp);
+ } else {
+ handleResponse((TGenerateDataPartitionTableHeartbeatResp) resp);
+ }
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableResp resp) {
+ updateResponse(resp);
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableHeartbeatResp resp) {
+ updateResponse(resp);
+ }
+
+ private <T> void updateResponse(T resp) {
+ if (currentGenerator == null) return;
+
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ setResponseFields(resp, DataPartitionTableGeneratorState.IN_PROGRESS.getCode(), String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100), RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case COMPLETED:
+ setResponseFields(resp, DataPartitionTableGeneratorState.SUCCESS.getCode(), "DataPartitionTable generation completed successfully", RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ case FAILED:
+ setResponseFields(resp, DataPartitionTableGeneratorState.FAILED.getCode(), "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage(), RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ LOGGER.info("DataPartitionTable generation failed with task ID: {}", currentTaskId);
+ break;
+ default:
+ setResponseFields(resp, DataPartitionTableGeneratorState.UNKNOWN.getCode(), "Unknown task status: " + currentGenerator.getStatus(), RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ LOGGER.info("DataPartitionTable generation failed with task ID: {}", currentTaskId);
+ break;
+ }
+ }
+
+ private <T> void setResponseFields(T resp, int errorCode, String message, TSStatus status) {
+ try {
+ Method setErrorCode = resp.getClass().getMethod("setErrorCode", int.class);
+ Method setMessage = resp.getClass().getMethod("setMessage", String.class);
+ Method setStatus = resp.getClass().getMethod("setStatus", TSStatus.class);
+
+ setErrorCode.invoke(resp, errorCode);
+ setMessage.invoke(resp, message);
+ setStatus.invoke(resp, status);
+ } catch (Exception e) {
+ LOGGER.error("Failed to set response fields", e);
+ }
+ }
+
+ /** Process data directory to find the earliest timeslots for each database. */
+ private void processDataDirectoryForEarliestTimeslots(
Review Comment:
Better log your conclusion, i.e., the earlist timeslot for each database.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (!TimePartitionUtils.satisfyPartitionId(localEarliestSlot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ }
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Flow requestPartitionTablesHeartBeat() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+
+ switch (state) {
+ case SUCCESS:
+ List<ByteBuffer> byteBufferList = resp.getDatabaseScopedDataPartitionTables();
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = deserializeDatabaseScopedTableList(byteBufferList);
+ dataPartitionTables.put(dataNodeId, databaseScopedDataPartitionTableList);
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ try {
+ Thread.sleep(HEART_BEAT_REQUEST_RATE);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.error("Error checking DataPartitionTable status due to thread interruption.");
+ }
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
Review Comment:
```suggestion
setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
return Flow.HAS_MORE_STATE;
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -1219,6 +1219,11 @@ public class IoTDBConfig {
private long maxObjectSizeInByte = 4 * 1024 * 1024 * 1024L;
+ /* Need use these parameters when repair data partition table */
+ private int partitionTableRecoverWorkerNum = 10;
+ // Rate limit set to 10 MB/s
+ private int partitionTableRecoverMaxReadBytesPerSecond = 10;
Review Comment:
MaxReadMBs instead of `Bytes`
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (!TimePartitionUtils.satisfyPartitionId(localEarliestSlot.getStartTime(), earliestTimeslot)) {
Review Comment:
Judge whether earliestTimeSlot.startTime < localEarliestSlot.startTime
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/TimePartitionUtils.java:
##########
@@ -134,6 +157,14 @@ public static boolean satisfyPartitionId(long startTime, long endTime, long part
return startPartition <= partitionId && endPartition >= partitionId;
}
+ public static boolean satisfyPartitionId(long startTime, long partitionId) {
Review Comment:
No need to update this class in our current implementation.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +430,11 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
+
+ // Must be lower than the RPC request timeout, in milliseconds
+ private static final long timeoutMs = 50000;
Review Comment:
DO NOT append member variables easily.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/service/DataNodeInternalRPCServiceImplTest.java:
##########
@@ -81,6 +83,8 @@
public class DataNodeInternalRPCServiceImplTest {
+ private static final Logger LOG =
Review Comment:
Why change this file?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
Review Comment:
Change this map into a local variable.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3138,349 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+ // Set default value
+ resp.setDatabaseScopedDataPartitionTables(Collections.emptyList());
+ try {
+ currentGeneratorFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ parseGenerationStatus(resp);
+ if (currentGenerator.getStatus().equals(DataPartitionTableGenerator.TaskStatus.COMPLETED)) {
+ boolean success = false;
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = new ArrayList<>();
+ Map<String, DataPartitionTable> dataPartitionTableMap = currentGenerator.getDatabasePartitionTableMap();
+ if (!dataPartitionTableMap.isEmpty()) {
+ for (Map.Entry<String, DataPartitionTable> entry : dataPartitionTableMap.entrySet()) {
+ String database = entry.getKey();
+ DataPartitionTable dataPartitionTable = entry.getValue();
+ if (!StringUtils.isEmpty(database) && dataPartitionTable != null) {
+ DatabaseScopedDataPartitionTable databaseScopedDataPartitionTable = new DatabaseScopedDataPartitionTable(database, dataPartitionTable);
+ databaseScopedDataPartitionTableList.add(databaseScopedDataPartitionTable);
+ success = true;
+ }
+ }
+ }
+
+ if (success) {
+ List<ByteBuffer> result = serializeDatabaseScopedTableList(databaseScopedDataPartitionTableList);
+ resp.setDatabaseScopedDataPartitionTables(result);
+
+ // Clear current generator
+ currentGenerator = null;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+ return resp;
+ }
+
+ private <T> void parseGenerationStatus(T resp) {
+ if (resp instanceof TGenerateDataPartitionTableResp) {
+ handleResponse((TGenerateDataPartitionTableResp) resp);
+ } else {
+ handleResponse((TGenerateDataPartitionTableHeartbeatResp) resp);
+ }
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableResp resp) {
+ updateResponse(resp);
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableHeartbeatResp resp) {
+ updateResponse(resp);
+ }
+
+ private <T> void updateResponse(T resp) {
+ if (currentGenerator == null) return;
+
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ setResponseFields(resp, DataPartitionTableGeneratorState.IN_PROGRESS.getCode(), String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
Review Comment:
It is good to left progress record. You'd better log it for retrieving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944096902
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+ byte[] empty = new byte[0];
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGenerator.startGeneration().get(timeoutMs, TimeUnit.MILLISECONDS);
+
+ if (currentGenerator != null) {
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage("DataPartitionTable generation interrupted");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ DataPartitionTable dataPartitionTable = currentGenerator.getDataPartitionTable();
+ if (dataPartitionTable != null) {
+ byte[] result = serializeDataPartitionTable(dataPartitionTable);
+ resp.setDataPartitionTable(result);
+ }
+
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ default:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ }
+
+ // Clear current generator
+ currentGenerator = null;
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+
+ try {
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ DataPartitionTableGenerator.TaskStatus status = currentGenerator.getStatus();
+
+ switch (status) {
+ case IN_PROGRESS:
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage(
+ String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100));
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case FAILED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ default:
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("Unknown task status: " + status);
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ /** Process data directory to find the earliest timeslots for each database. */
+ private void processDataDirectoryForEarliestTimeslots(
+ File dataDir, Map<String, Long> earliestTimeslots) {
+ try {
+ Files.list(dataDir.toPath())
+ .filter(Files::isDirectory)
+ .forEach(
+ sequenceTypePath -> {
+ try {
+ Files.list(sequenceTypePath)
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944133288
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
Review Comment:
Choose another plan to change yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957696711
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+ // Set default value
+ resp.setDatabaseScopedDataPartitionTables(Collections.emptyList());
+ try {
+ currentGeneratorFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ parseGenerationStatus(resp);
+ if (currentGenerator.getStatus().equals(DataPartitionTableGenerator.TaskStatus.COMPLETED)) {
+ boolean success = false;
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = new ArrayList<>();
+ Map<String, DataPartitionTable> dataPartitionTableMap = currentGenerator.getDatabasePartitionTableMap();
+ if (!dataPartitionTableMap.isEmpty()) {
+ for (Map.Entry<String, DataPartitionTable> entry : dataPartitionTableMap.entrySet()) {
+ String database = entry.getKey();
+ DataPartitionTable dataPartitionTable = entry.getValue();
+ if (!StringUtils.isEmpty(database) && dataPartitionTable != null) {
+ DatabaseScopedDataPartitionTable databaseScopedDataPartitionTable = new DatabaseScopedDataPartitionTable(database, dataPartitionTable);
+ databaseScopedDataPartitionTableList.add(databaseScopedDataPartitionTable);
+ success = true;
+ }
+ }
+ }
+
+ if (success) {
+ List<ByteBuffer> result = serializeDatabaseScopedTableList(databaseScopedDataPartitionTableList);
+ resp.setDatabaseScopedDataPartitionTables(result);
+
+ // Clear current generator
+ currentGenerator = null;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+ return resp;
+ }
+
+ private <T> void parseGenerationStatus(T resp) {
+ if (resp instanceof TGenerateDataPartitionTableResp) {
+ handleResponse((TGenerateDataPartitionTableResp) resp);
+ } else {
+ handleResponse((TGenerateDataPartitionTableHeartbeatResp) resp);
+ }
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableResp resp) {
+ updateResponse(resp);
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableHeartbeatResp resp) {
+ updateResponse(resp);
+ }
+
+ private <T> void updateResponse(T resp) {
+ if (currentGenerator == null) return;
+
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ setResponseFields(resp, DataPartitionTableGeneratorState.IN_PROGRESS.getCode(), String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100), RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case COMPLETED:
+ setResponseFields(resp, DataPartitionTableGeneratorState.SUCCESS.getCode(), "DataPartitionTable generation completed successfully", RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ case FAILED:
+ setResponseFields(resp, DataPartitionTableGeneratorState.FAILED.getCode(), "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage(), RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ LOGGER.info("DataPartitionTable generation failed with task ID: {}", currentTaskId);
+ break;
+ default:
+ setResponseFields(resp, DataPartitionTableGeneratorState.UNKNOWN.getCode(), "Unknown task status: " + currentGenerator.getStatus(), RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ LOGGER.info("DataPartitionTable generation failed with task ID: {}", currentTaskId);
+ break;
+ }
+ }
+
+ private <T> void setResponseFields(T resp, int errorCode, String message, TSStatus status) {
+ try {
+ Method setErrorCode = resp.getClass().getMethod("setErrorCode", int.class);
+ Method setMessage = resp.getClass().getMethod("setMessage", String.class);
+ Method setStatus = resp.getClass().getMethod("setStatus", TSStatus.class);
+
+ setErrorCode.invoke(resp, errorCode);
+ setMessage.invoke(resp, message);
+ setStatus.invoke(resp, status);
+ } catch (Exception e) {
+ LOGGER.error("Failed to set response fields", e);
+ }
+ }
+
+ /** Process data directory to find the earliest timeslots for each database. */
+ private void processDataDirectoryForEarliestTimeslots(
+ File dataDir, Map<String, Long> earliestTimeslots) {
+ Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
+ try (Stream<Path> sequenceTypePaths = Files.list(dataDir.toPath())) {
+ sequenceTypePaths
+ .filter(Files::isDirectory)
+ .forEach(
+ sequenceTypePath -> {
+ try (Stream<Path> dbPaths = Files.list(sequenceTypePath)) {
+ dbPaths
+ .filter(Files::isDirectory)
+ .forEach(
+ dbPath -> {
+ String databaseName = dbPath.getFileName().toString();
+ if (DataPartitionTableGenerator.IGNORE_DATABASE.contains(
+ databaseName)) {
+ return;
+ }
+ databaseEarliestRegionMap.computeIfAbsent(
+ databaseName, key -> Long.MAX_VALUE);
+ long earliestTimeslot = findEarliestTimeslotInDatabase(dbPath.toFile(), databaseEarliestRegionMap);
+
+ if (earliestTimeslot != Long.MAX_VALUE) {
+ earliestTimeslots.merge(databaseName, earliestTimeslot, Math::min);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error(
+ "Failed to process data directory: {}", sequenceTypePath.toFile(), e);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to process data directory: {}", dataDir, e);
+ }
+ }
+
+ /** Find the earliest timeslot in a database directory. */
+ private long findEarliestTimeslotInDatabase(File databaseDir, Map<String, Long> databaseEarliestRegionMap) {
+ String databaseName = databaseDir.getName();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ try (Stream<Path> databasePaths = Files.list(databaseDir.toPath())) {
+ databasePaths
+ .filter(Files::isDirectory)
+ .forEach(
+ regionPath -> {
+ Future<?> future =
+ findEarliestTimeSlotExecutor.submit(
+ () -> {
+ try (Stream<Path> regionPaths = Files.list(regionPath)) {
+ regionPaths
+ .filter(Files::isDirectory)
+ .forEach(
+ timeSlotPath -> {
+ try {
+ Optional<Path> matchedFile =
+ Files.find(
+ timeSlotPath,
+ 1,
+ (path, attrs) ->
+ attrs.isRegularFile()
+ && path.toString()
+ .endsWith(
+ DataPartitionTableGenerator
+ .SCAN_FILE_SUFFIX_NAME))
+ .findFirst();
+ if (!matchedFile.isPresent()) {
+ return;
+ }
+ String timeSlotName = timeSlotPath.getFileName().toString();
+ long timeslot = Long.parseLong(timeSlotName);
+ databaseEarliestRegionMap.compute(
+ databaseName,
+ (k, v) ->
+ v == null ? timeslot : Math.min(v, timeslot));
+ } catch (IOException e) {
+ LOGGER.error(
+ "Failed to find any {} files in the {} directory",
+ DataPartitionTableGenerator.SCAN_FILE_SUFFIX_NAME,
+ timeSlotPath,
+ e);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to scan {}", regionPath, e);
+ }
+ });
+ futureList.add(future);
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to walk database directory: {}", databaseDir, e);
+ }
+
+ for (Future<?> future : futureList) {
+ try {
+ future.get();
+ } catch (InterruptedException | ExecutionException e) {
+ LOGGER.error("Failed to wait for task completion", e);
+ Thread.currentThread().interrupt();
+ }
+ }
+ findEarliestTimeSlotExecutor.shutdownNow();
+ return databaseEarliestRegionMap.get(databaseName);
+ }
+
+ private List<ByteBuffer> serializeDatabaseScopedTableList(List<DatabaseScopedDataPartitionTable> list) {
+ if (list == null || list.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ List<ByteBuffer> result = new ArrayList<>(list.size());
+
+ for (DatabaseScopedDataPartitionTable table : list) {
+ try (PublicBAOS baos = new PublicBAOS();
+ DataOutputStream oos = new DataOutputStream(baos)) {
+
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ table.serialize(oos, protocol);
+
+ result.add(ByteBuffer.wrap(baos.toByteArray()));
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957305510
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944161688
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/ConfigNodeProcedureEnv.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.confignode.manager.ConfigManager;
+
+/**
+ * Environment object for ConfigNode procedures. Provides access to ConfigManager and other
+ * necessary components.
+ */
+public class ConfigNodeProcedureEnv {
+
+ private final ConfigManager configManager;
+
+ public ConfigNodeProcedureEnv(ConfigManager configManager) {
+ this.configManager = configManager;
+ }
+
+ public ConfigManager getConfigManager() {
+ return configManager;
+ }
Review Comment:
ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944136067
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -4367,4 +4371,20 @@ public long getMaxObjectSizeInByte() {
public void setMaxObjectSizeInByte(long maxObjectSizeInByte) {
this.maxObjectSizeInByte = maxObjectSizeInByte;
}
+
+ public int getPartitionTableRecoverWorkerNum() {
+ return partitionTableRecoverWorkerNum;
+ }
+
+ public void setPartitionTableRecoverWorkerNum(int partitionTableRecoverWorkerNum) {
+ this.partitionTableRecoverWorkerNum = partitionTableRecoverWorkerNum;
+ }
+
+ public int getPartitionTableRecoverMaxReadBytesPerSecond() {
+ return partitionTableRecoverMaxReadBytesPerSecond;
+ }
+
+ public void setPartitionTableRecoverMaxReadBytesPerSecond(int partitionTableRecoverWorkerNum) {
+ this.partitionTableRecoverWorkerNum = partitionTableRecoverWorkerNum;
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944075438
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
Review Comment:
Unnecessarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957422826
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupType;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.partition.executor.SeriesPartitionExecutor;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.commons.utils.rateLimiter.LeakyBucketRateLimiter;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Generator for DataPartitionTable by scanning tsfile resources. This class scans the data
+ * directory structure and builds a complete DataPartitionTable based on existing tsfiles.
+ */
+public class DataPartitionTableGenerator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DataPartitionTableGenerator.class);
+
+ // Task status
+ private volatile TaskStatus status = TaskStatus.NOT_STARTED;
+ private volatile String errorMessage;
+ private Map<String, DataPartitionTable> databasePartitionTableMap = new ConcurrentHashMap<>();
+
+ // Progress tracking
+ private final AtomicInteger processedFiles = new AtomicInteger(0);
+ private final AtomicInteger failedFiles = new AtomicInteger(0);
+ private final AtomicLong totalFiles = new AtomicLong(0);
+
+ // Configuration
+ private final ExecutorService executor;
+ private final Set<String> databases;
+ private final int seriesSlotNum;
+ private final String seriesPartitionExecutorClass;
+
+ private final LeakyBucketRateLimiter limiter =
+ new LeakyBucketRateLimiter(
+ (long)
+ IoTDBDescriptor.getInstance()
+ .getConfig()
+ .getPartitionTableRecoverMaxReadBytesPerSecond()
+ * 1024
+ * 1024);
+
+ public static final Set<String> IGNORE_DATABASE =
+ new HashSet<String>() {
+ {
+ add("root.__audit");
+ add("root.__system");
+ }
+ };
+
+ public static final String SCAN_FILE_SUFFIX_NAME = ".tsfile";
+
+ public DataPartitionTableGenerator(
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public Map<String, DataPartitionTable> getDatabasePartitionTableMap() {
+ return databasePartitionTableMap;
+ }
+
+ public enum TaskStatus {
+ NOT_STARTED,
+ IN_PROGRESS,
+ COMPLETED,
+ FAILED
+ }
+
+ /** Start generating DataPartitionTable asynchronously. */
+ public CompletableFuture<Void> startGeneration() {
+ if (status != TaskStatus.NOT_STARTED) {
+ throw new IllegalStateException("Task is already started or completed");
+ }
+
+ status = TaskStatus.IN_PROGRESS;
+ return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);
+ }
+
+ private void generateDataPartitionTableByMemory() {
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap = new ConcurrentHashMap<>();
+ List<CompletableFuture<Void>> futures = new ArrayList<>();
+
+ SeriesPartitionExecutor seriesPartitionExecutor =
+ SeriesPartitionExecutor.getSeriesPartitionExecutor(
+ seriesPartitionExecutorClass, seriesSlotNum);
+
+ try {
+ for (DataRegion dataRegion : StorageEngine.getInstance().getAllDataRegions()) {
+ CompletableFuture<Void> regionFuture =
+ CompletableFuture.runAsync(
+ () -> {
+ try {
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ String databaseName = dataRegion.getDatabaseName();
+ if (!databases.contains(databaseName)
+ || IGNORE_DATABASE.contains(databaseName)) {
+ return;
+ }
+
+ tsFileManager.readLock();
+ List<TsFileResource> seqTsFileList = tsFileManager.getTsFileList(true);
+ List<TsFileResource> unseqTsFileList = tsFileManager.getTsFileList(false);
+ tsFileManager.readUnlock();
+
+ constructDataPartitionMap(
+ seqTsFileList, seriesPartitionExecutor, dataPartitionMap);
+ constructDataPartitionMap(
+ unseqTsFileList, seriesPartitionExecutor, dataPartitionMap);
+
+ if (dataPartitionMap.isEmpty()) {
+ LOG.error("Failed to generate DataPartitionTable, dataPartitionMap is empty");
+ status = TaskStatus.FAILED;
+ errorMessage = "DataPartitionMap is empty after processing resource file";
+ return;
+ }
+
+ DataPartitionTable dataPartitionTable = new DataPartitionTable(dataPartitionMap);
+
+ databasePartitionTableMap.compute(databaseName, (k,v) -> {
+ if (v == null) {
+ return new DataPartitionTable(dataPartitionMap);
+ }
+ v.merge(dataPartitionTable);
+ return v;
+ });
+ } catch (Exception e) {
+ LOG.error("Error processing data region: {}", dataRegion.getDatabaseName(), e);
+ failedFiles.incrementAndGet();
+ errorMessage = "Failed to process data region: " + e.getMessage();
+ }
+ },
+ executor);
+ futures.add(regionFuture);
+ }
+
+ // Wait for all tasks to complete
+ CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
+
+ status = TaskStatus.COMPLETED;
+ LOG.info(
+ "DataPartitionTable generation completed successfully. Processed: {}, Failed: {}",
+ processedFiles.get(),
+ failedFiles.get());
+ } catch (Exception e) {
+ LOG.error("Failed to generate DataPartitionTable", e);
+ status = TaskStatus.FAILED;
+ errorMessage = "Generation failed: " + e.getMessage();
+ }
+ }
+
+ private void constructDataPartitionMap(
+ List<TsFileResource> seqTsFileList,
+ SeriesPartitionExecutor seriesPartitionExecutor,
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap) {
+ for (TsFileResource tsFileResource : seqTsFileList) {
+ try {
+ Set<IDeviceID> devices = tsFileResource.getDevices(limiter);
+ long timeSlotId = tsFileResource.getTsFileID().timePartitionId;
+ int regionId = tsFileResource.getTsFileID().regionId;
+
+ TConsensusGroupId consensusGroupId = new TConsensusGroupId();
+ consensusGroupId.setId(regionId);
+ consensusGroupId.setType(TConsensusGroupType.DataRegion);
+
+ for (IDeviceID deviceId : devices) {
+ TSeriesPartitionSlot seriesSlotId =
+ seriesPartitionExecutor.getSeriesPartitionSlot(deviceId);
+ TTimePartitionSlot timePartitionSlot =
+ new TTimePartitionSlot(TimePartitionUtils.getTimeByPartitionId(timeSlotId));
+ dataPartitionMap
+ .computeIfAbsent(
+ seriesSlotId, empty -> newSeriesPartitionTable(consensusGroupId, timeSlotId))
+ .putDataPartition(timePartitionSlot, consensusGroupId);
+ }
+ processedFiles.incrementAndGet();
+ } catch (Exception e) {
+ failedFiles.incrementAndGet();
+ LOG.error("Failed to process tsfile {}, {}", tsFileResource.getTsFileID(), e.getMessage());
+ }
+ }
+ }
+
+ private static SeriesPartitionTable newSeriesPartitionTable(
+ TConsensusGroupId consensusGroupId, long timeSlotId) {
+ SeriesPartitionTable seriesPartitionTable = new SeriesPartitionTable();
+ TTimePartitionSlot timePartitionSlot =
+ new TTimePartitionSlot(TimePartitionUtils.getTimeByPartitionId(timeSlotId));
+ seriesPartitionTable.putDataPartition(timePartitionSlot, consensusGroupId);
+ return seriesPartitionTable;
+ }
+
+ // Getters
+ public TaskStatus getStatus() {
+ return status;
+ }
+
+ public String getErrorMessage() {
+ return errorMessage;
+ }
+
+ public double getProgress() {
+ if (totalFiles.get() == 0) {
+ return 0.0;
+ }
+ return (double) (processedFiles.get() + failedFiles.get()) / totalFiles.get();
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944136067
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -4367,4 +4371,20 @@ public long getMaxObjectSizeInByte() {
public void setMaxObjectSizeInByte(long maxObjectSizeInByte) {
this.maxObjectSizeInByte = maxObjectSizeInByte;
}
+
+ public int getPartitionTableRecoverWorkerNum() {
+ return partitionTableRecoverWorkerNum;
+ }
+
+ public void setPartitionTableRecoverWorkerNum(int partitionTableRecoverWorkerNum) {
+ this.partitionTableRecoverWorkerNum = partitionTableRecoverWorkerNum;
+ }
+
+ public int getPartitionTableRecoverMaxReadBytesPerSecond() {
+ return partitionTableRecoverMaxReadBytesPerSecond;
+ }
+
+ public void setPartitionTableRecoverMaxReadBytesPerSecond(int partitionTableRecoverWorkerNum) {
+ this.partitionTableRecoverWorkerNum = partitionTableRecoverWorkerNum;
Review Comment:
Unnecessarily, ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944031281
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -110,6 +116,13 @@ public class ConfigNode extends ServerCommandLine implements ConfigNodeMBean {
private int exitStatusCode = 0;
+ private Future<Void> dataPartitionTableCheckFuture;
+
+ private ExecutorService dataPartitionTableCheckExecutor =
+ IoTDBThreadPoolFactory.newSingleThreadExecutor("DATA_PARTITION_TABLE_CHECK");
Review Comment:
The executor will be shut down after it have done:
<img width="1070" height="733" alt="image" src="https://github.com/user-attachments/assets/1dbc509a-7867-40b2-a909-c6b8dbf7cef3" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944893763
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/service/DataNodeInternalRPCServiceImplTest.java:
##########
@@ -81,6 +83,8 @@
public class DataNodeInternalRPCServiceImplTest {
+ private static final Logger LOG =
Review Comment:
I forgot to clean it. I coded some unit tests for test rpc interfaces on the local ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2963720324
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
Review Comment:
How about restarting an empty cluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944075438
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
Review Comment:
Unnecessarily, ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944135473
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java:
##########
@@ -1374,6 +1375,16 @@ public TSStatus createRegionGroups(
}
}
+ /** Used to repair the lost data partition table */
+ public TSStatus dataPartitionTableIntegrityCheck() {
+ DataPartitionTableIntegrityCheckProcedure procedure;
+ synchronized (this) {
+ procedure = new DataPartitionTableIntegrityCheckProcedure();
+ executor.submitProcedure(procedure);
+ }
+ return waitingProcedureFinished(procedure);
Review Comment:
Unnecessarily, ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944646631
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
Review Comment:
Unnecessarily, ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957304549
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957418819
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -0,0 +1,254 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupType;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.partition.executor.SeriesPartitionExecutor;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.commons.utils.rateLimiter.LeakyBucketRateLimiter;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Generator for DataPartitionTable by scanning tsfile resources. This class scans the data
+ * directory structure and builds a complete DataPartitionTable based on existing tsfiles.
+ */
+public class DataPartitionTableGenerator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DataPartitionTableGenerator.class);
+
+ // Task status
+ private volatile TaskStatus status = TaskStatus.NOT_STARTED;
+ private volatile String errorMessage;
+ private Map<String, DataPartitionTable> databasePartitionTableMap = new ConcurrentHashMap<>();
+
+ // Progress tracking
+ private final AtomicInteger processedFiles = new AtomicInteger(0);
+ private final AtomicInteger failedFiles = new AtomicInteger(0);
+ private final AtomicLong totalFiles = new AtomicLong(0);
+
+ // Configuration
+ private final ExecutorService executor;
+ private final Set<String> databases;
+ private final int seriesSlotNum;
+ private final String seriesPartitionExecutorClass;
+
+ private final LeakyBucketRateLimiter limiter =
+ new LeakyBucketRateLimiter(
+ (long)
+ IoTDBDescriptor.getInstance()
+ .getConfig()
+ .getPartitionTableRecoverMaxReadBytesPerSecond()
+ * 1024
+ * 1024);
+
+ public static final Set<String> IGNORE_DATABASE =
+ new HashSet<String>() {
+ {
+ add("root.__audit");
+ add("root.__system");
+ }
+ };
+
+ public static final String SCAN_FILE_SUFFIX_NAME = ".tsfile";
+
+ public DataPartitionTableGenerator(
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public Map<String, DataPartitionTable> getDatabasePartitionTableMap() {
+ return databasePartitionTableMap;
+ }
+
+ public enum TaskStatus {
+ NOT_STARTED,
+ IN_PROGRESS,
+ COMPLETED,
+ FAILED
+ }
+
+ /** Start generating DataPartitionTable asynchronously. */
+ public CompletableFuture<Void> startGeneration() {
+ if (status != TaskStatus.NOT_STARTED) {
+ throw new IllegalStateException("Task is already started or completed");
+ }
+
+ status = TaskStatus.IN_PROGRESS;
+ return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);
+ }
+
+ private void generateDataPartitionTableByMemory() {
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap = new ConcurrentHashMap<>();
+ List<CompletableFuture<Void>> futures = new ArrayList<>();
+
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2954309084
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (localEarliestSlot.getStartTime() > TimePartitionUtils.getTimeByPartitionId(earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ }
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Flow requestPartitionTablesHeartBeat() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+
+ switch (state) {
+ case SUCCESS:
+ List<ByteBuffer> byteBufferList = resp.getDatabaseScopedDataPartitionTables();
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = deserializeDatabaseScopedTableList(byteBufferList);
+ dataPartitionTables.put(dataNodeId, databaseScopedDataPartitionTableList);
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ try {
+ Thread.sleep(HEART_BEAT_REQUEST_RATE);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.error("Error checking DataPartitionTable status due to thread interruption.");
+ }
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+
+ dataPartitionTables.forEach((k, v) -> v.forEach(databaseScopedDataPartitionTable -> {
+ if (!databaseScopedDataPartitionTable.getDatabase().equals(database)) {
+ return;
+ }
+ finalDataPartitionTables.put(database, new DataPartitionTable(finalDataPartitionMap).merge(databaseScopedDataPartitionTable.getDataPartitionTable()));
+ }));
+ }
+
+ LOG.info("DataPartitionTables merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "CRZbulabula (via GitHub)" <gi...@apache.org>.
CRZbulabula commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2929683212
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
Review Comment:
Should be processed asynchronosly
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
Review Comment:
This could be set to a local variable.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -110,6 +116,13 @@ public class ConfigNode extends ServerCommandLine implements ConfigNodeMBean {
private int exitStatusCode = 0;
+ private Future<Void> dataPartitionTableCheckFuture;
+
+ private ExecutorService dataPartitionTableCheckExecutor =
+ IoTDBThreadPoolFactory.newSingleThreadExecutor("DATA_PARTITION_TABLE_CHECK");
+
+ private final CountDownLatch latch = new CountDownLatch(1);
Review Comment:
This CountDownLatch is unnecessary.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ // Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeout());
+
+ while (latch.getCount() > 0) {
+ List<Integer> dnList =
+ configManager
+ .getLoadManager()
+ .filterDataNodeThroughStatus(NodeStatus.Running);
+ if (dnList != null && !dnList.isEmpty()) {
+ LOGGER.info("Starting dataPartitionTableIntegrityCheck...");
+ TSStatus status =
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOGGER.error("Data partition table integrity check failed!");
+ }
+ latch.countDown();
+ } else {
+ LOGGER.info("No running datanodes found, waiting...");
+ Thread.sleep(5000); // 等待5秒后重新检查
Review Comment:
```suggestion
Thread.sleep(5000);
```
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java:
##########
@@ -352,6 +352,9 @@ public DataSet registerDataNode(TDataNodeRegisterReq req) {
// Adjust the maximum RegionGroup number of each Database
getClusterSchemaManager().adjustMaxRegionGroupNum();
+ // Check if all DataNodes are registered and trigger integrity check if needed
+ checkAndTriggerIntegrityCheck();
Review Comment:
Remove this function.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ // Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeout());
Review Comment:
Remove this.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
Review Comment:
Rectify these codes. The only thing u need to check is whether the database's earlist timeSlot greater than the DataNode's.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
Review Comment:
Should extract some common functions for DataPartitionTable, SeriesPartitionTable.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -110,6 +116,13 @@ public class ConfigNode extends ServerCommandLine implements ConfigNodeMBean {
private int exitStatusCode = 0;
+ private Future<Void> dataPartitionTableCheckFuture;
+
+ private ExecutorService dataPartitionTableCheckExecutor =
+ IoTDBThreadPoolFactory.newSingleThreadExecutor("DATA_PARTITION_TABLE_CHECK");
Review Comment:
Verify the lifecycle of this variable, it should be cleaned after your check passed.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
Review Comment:
DO NOT use _terminating_, this is ambiguous here.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
Review Comment:
Better to append a prefix for all of your logs appended, e.g.,
```suggestion
"[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
```
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/conf/ConfigNodeConfig.java:
##########
@@ -319,6 +319,8 @@ public class ConfigNodeConfig {
private long forceWalPeriodForConfigNodeSimpleInMs = 100;
+ private long partitionTableRecoverWaitAllDnUpTimeout = 60000;
Review Comment:
Refactor this parameter
```suggestion
private long partitionTableRecoverWaitAllDnUpTimeoutInMs = 60000;
```
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
Review Comment:
HashSet<>(allDataNodes).containsAll(failedDataNodes) is always true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944031281
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -110,6 +116,13 @@ public class ConfigNode extends ServerCommandLine implements ConfigNodeMBean {
private int exitStatusCode = 0;
+ private Future<Void> dataPartitionTableCheckFuture;
+
+ private ExecutorService dataPartitionTableCheckExecutor =
+ IoTDBThreadPoolFactory.newSingleThreadExecutor("DATA_PARTITION_TABLE_CHECK");
Review Comment:
The executor will be shut down after it have done.
`protected void start() throws IoTDBException {
try {
// Do ConfigNode startup checks
LOGGER.info("Starting IoTDB {}", IoTDBConstant.VERSION_WITH_BUILD);
ConfigNodeStartupCheck checks = new ConfigNodeStartupCheck(IoTDBConstant.CN_ROLE);
checks.startUpCheck();
} catch (StartupException | ConfigurationException | IOException e) {
LOGGER.error("Meet error when doing start checking", e);
throw new IoTDBException("Error starting", -1);
}
active();
LOGGER.info("IoTDB started");
if (dataPartitionTableCheckFuture != null) {
try {
dataPartitionTableCheckFuture.get();
} catch (ExecutionException | InterruptedException e) {
LOGGER.error("Data partition table check task execute failed", e);
} finally {
dataPartitionTableCheckExecutor.shutdownNow();
}
}
}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2954304185
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957384287
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+ // Set default value
+ resp.setDatabaseScopedDataPartitionTables(Collections.emptyList());
+ try {
+ currentGeneratorFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ parseGenerationStatus(resp);
+ if (currentGenerator.getStatus().equals(DataPartitionTableGenerator.TaskStatus.COMPLETED)) {
+ boolean success = false;
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = new ArrayList<>();
+ Map<String, DataPartitionTable> dataPartitionTableMap = currentGenerator.getDatabasePartitionTableMap();
+ if (!dataPartitionTableMap.isEmpty()) {
+ for (Map.Entry<String, DataPartitionTable> entry : dataPartitionTableMap.entrySet()) {
+ String database = entry.getKey();
+ DataPartitionTable dataPartitionTable = entry.getValue();
+ if (!StringUtils.isEmpty(database) && dataPartitionTable != null) {
+ DatabaseScopedDataPartitionTable databaseScopedDataPartitionTable = new DatabaseScopedDataPartitionTable(database, dataPartitionTable);
+ databaseScopedDataPartitionTableList.add(databaseScopedDataPartitionTable);
+ success = true;
+ }
+ }
+ }
+
+ if (success) {
+ List<ByteBuffer> result = serializeDatabaseScopedTableList(databaseScopedDataPartitionTableList);
+ resp.setDatabaseScopedDataPartitionTables(result);
+
+ // Clear current generator
+ currentGenerator = null;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+ return resp;
+ }
+
+ private <T> void parseGenerationStatus(T resp) {
+ if (resp instanceof TGenerateDataPartitionTableResp) {
+ handleResponse((TGenerateDataPartitionTableResp) resp);
+ } else {
+ handleResponse((TGenerateDataPartitionTableHeartbeatResp) resp);
+ }
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableResp resp) {
+ updateResponse(resp);
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableHeartbeatResp resp) {
+ updateResponse(resp);
+ }
Review Comment:
> What is the point of the encapsulation
Because the generics.
<img width="1230" height="794" alt="20260319-102009" src="https://github.com/user-attachments/assets/eba6eb14-0709-4fa4-9581-76594b0e7d5d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2958180177
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+ // Set default value
+ resp.setDatabaseScopedDataPartitionTables(Collections.emptyList());
+ try {
+ currentGeneratorFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ parseGenerationStatus(resp);
+ if (currentGenerator.getStatus().equals(DataPartitionTableGenerator.TaskStatus.COMPLETED)) {
+ boolean success = false;
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = new ArrayList<>();
+ Map<String, DataPartitionTable> dataPartitionTableMap = currentGenerator.getDatabasePartitionTableMap();
+ if (!dataPartitionTableMap.isEmpty()) {
+ for (Map.Entry<String, DataPartitionTable> entry : dataPartitionTableMap.entrySet()) {
+ String database = entry.getKey();
+ DataPartitionTable dataPartitionTable = entry.getValue();
+ if (!StringUtils.isEmpty(database) && dataPartitionTable != null) {
+ DatabaseScopedDataPartitionTable databaseScopedDataPartitionTable = new DatabaseScopedDataPartitionTable(database, dataPartitionTable);
+ databaseScopedDataPartitionTableList.add(databaseScopedDataPartitionTable);
+ success = true;
+ }
+ }
+ }
+
+ if (success) {
+ List<ByteBuffer> result = serializeDatabaseScopedTableList(databaseScopedDataPartitionTableList);
+ resp.setDatabaseScopedDataPartitionTables(result);
+
+ // Clear current generator
+ currentGenerator = null;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+ return resp;
+ }
+
+ private <T> void parseGenerationStatus(T resp) {
+ if (resp instanceof TGenerateDataPartitionTableResp) {
+ handleResponse((TGenerateDataPartitionTableResp) resp);
+ } else {
+ handleResponse((TGenerateDataPartitionTableHeartbeatResp) resp);
+ }
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableResp resp) {
+ updateResponse(resp);
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableHeartbeatResp resp) {
+ updateResponse(resp);
+ }
+
+ private <T> void updateResponse(T resp) {
+ if (currentGenerator == null) return;
+
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ setResponseFields(resp, DataPartitionTableGeneratorState.IN_PROGRESS.getCode(), String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100), RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case COMPLETED:
+ setResponseFields(resp, DataPartitionTableGeneratorState.SUCCESS.getCode(), "DataPartitionTable generation completed successfully", RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ case FAILED:
+ setResponseFields(resp, DataPartitionTableGeneratorState.FAILED.getCode(), "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage(), RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ LOGGER.info("DataPartitionTable generation failed with task ID: {}", currentTaskId);
+ break;
+ default:
+ setResponseFields(resp, DataPartitionTableGeneratorState.UNKNOWN.getCode(), "Unknown task status: " + currentGenerator.getStatus(), RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ LOGGER.info("DataPartitionTable generation failed with task ID: {}", currentTaskId);
+ break;
+ }
+ }
+
+ private <T> void setResponseFields(T resp, int errorCode, String message, TSStatus status) {
+ try {
+ Method setErrorCode = resp.getClass().getMethod("setErrorCode", int.class);
+ Method setMessage = resp.getClass().getMethod("setMessage", String.class);
+ Method setStatus = resp.getClass().getMethod("setStatus", TSStatus.class);
+
+ setErrorCode.invoke(resp, errorCode);
+ setMessage.invoke(resp, message);
+ setStatus.invoke(resp, status);
+ } catch (Exception e) {
+ LOGGER.error("Failed to set response fields", e);
+ }
+ }
Review Comment:
> Avoid using reflecting until totally necessary
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944652824
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupType;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.partition.executor.SeriesPartitionExecutor;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.commons.utils.rateLimiter.LeakyBucketRateLimiter;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Generator for DataPartitionTable by scanning tsfile resources. This class scans the data
+ * directory structure and builds a complete DataPartitionTable based on existing tsfiles.
+ */
+public class DataPartitionTableGenerator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DataPartitionTableGenerator.class);
+
+ // Task status
+ private volatile TaskStatus status = TaskStatus.NOT_STARTED;
+ private volatile String errorMessage;
+ private volatile DataPartitionTable dataPartitionTable;
+
+ // Progress tracking
+ private final AtomicInteger processedFiles = new AtomicInteger(0);
+ private final AtomicInteger failedFiles = new AtomicInteger(0);
+ private final AtomicLong totalFiles = new AtomicLong(0);
+
+ // Configuration
+ private String[] dataDirectories;
+ private final ExecutorService executor;
+ private final Set<String> databases;
+ private final int seriesSlotNum;
+ private final String seriesPartitionExecutorClass;
+
+ private static final int EXECUTOR_MAX_TIMEOUT = 60;
+
+ private static final LeakyBucketRateLimiter limiter =
+ new LeakyBucketRateLimiter(
+ (long)
+ IoTDBDescriptor.getInstance()
+ .getConfig()
+ .getPartitionTableRecoverMaxReadBytesPerSecond()
+ * 1024
+ * 1024);
+
+ public static final String SCAN_FILE_SUFFIX_NAME = ".tsfile";
+ public static final Set<String> IGNORE_DATABASE =
+ new HashSet<String>() {
+ {
+ add("root.__audit");
+ add("root.__system");
+ }
+ };
+
+ public DataPartitionTableGenerator(
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String dataDirectory,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = new String[] {dataDirectory};
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String[] dataDirectories,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = dataDirectories;
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public enum TaskStatus {
+ NOT_STARTED,
+ IN_PROGRESS,
+ COMPLETED,
+ FAILED
+ }
+
+ /** Start generating DataPartitionTable asynchronously. */
+ public CompletableFuture<Void> startGeneration() {
+ if (status != TaskStatus.NOT_STARTED) {
+ throw new IllegalStateException("Task is already started or completed");
+ }
+
+ status = TaskStatus.IN_PROGRESS;
+ return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);
+ }
+
+ private void generateDataPartitionTableByMemory() {
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap = new ConcurrentHashMap<>();
+ List<CompletableFuture<Void>> futures = new ArrayList<>();
+
+ SeriesPartitionExecutor seriesPartitionExecutor =
+ SeriesPartitionExecutor.getSeriesPartitionExecutor(
+ seriesPartitionExecutorClass, seriesSlotNum);
+
+ for (DataRegion dataRegion : StorageEngine.getInstance().getAllDataRegions()) {
+ CompletableFuture<Void> regionFuture =
+ CompletableFuture.runAsync(
+ () -> {
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ String databaseName = dataRegion.getDatabaseName();
+ if (!databases.contains(databaseName) || IGNORE_DATABASE.contains(databaseName)) {
+ return;
+ }
+
+ tsFileManager.readLock();
+ List<TsFileResource> seqTsFileList = tsFileManager.getTsFileList(true);
+ List<TsFileResource> unseqTsFileList = tsFileManager.getTsFileList(false);
+ tsFileManager.readUnlock();
+
+ constructDataPartitionMap(seqTsFileList, seriesPartitionExecutor, dataPartitionMap);
+ constructDataPartitionMap(
+ unseqTsFileList, seriesPartitionExecutor, dataPartitionMap);
+ },
+ executor);
+ futures.add(regionFuture);
+ }
+
+ // Wait for all tasks to complete
+ CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
+
+ if (dataPartitionMap.isEmpty()) {
+ LOG.error("Failed to generate DataPartitionTable, dataPartitionMap is empty");
+ status = TaskStatus.FAILED;
+ return;
+ }
+
+ dataPartitionTable = new DataPartitionTable(dataPartitionMap);
+ status = TaskStatus.COMPLETED;
+ }
+
+ private static void constructDataPartitionMap(
+ List<TsFileResource> seqTsFileList,
+ SeriesPartitionExecutor seriesPartitionExecutor,
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap) {
+ for (TsFileResource tsFileResource : seqTsFileList) {
+ Set<IDeviceID> devices = tsFileResource.getDevices(limiter);
+ long timeSlotId = tsFileResource.getTsFileID().timePartitionId;
+ int regionId = tsFileResource.getTsFileID().regionId;
+
+ TConsensusGroupId consensusGroupId = new TConsensusGroupId();
+ consensusGroupId.setId(regionId);
+ consensusGroupId.setType(TConsensusGroupType.DataRegion);
+
+ for (IDeviceID deviceId : devices) {
+ TSeriesPartitionSlot seriesSlotId =
+ seriesPartitionExecutor.getSeriesPartitionSlot(deviceId);
+ TTimePartitionSlot timePartitionSlot =
+ new TTimePartitionSlot(TimePartitionUtils.getTimeByPartitionId(timeSlotId));
+ dataPartitionMap
+ .computeIfAbsent(
+ seriesSlotId, empty -> newSeriesPartitionTable(consensusGroupId, timeSlotId))
Review Comment:
It's wrong, ignore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944661405
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java:
##########
@@ -352,6 +352,9 @@ public DataSet registerDataNode(TDataNodeRegisterReq req) {
// Adjust the maximum RegionGroup number of each Database
getClusterSchemaManager().adjustMaxRegionGroupNum();
+ // Check if all DataNodes are registered and trigger integrity check if needed
+ checkAndTriggerIntegrityCheck();
+
Review Comment:
checkAndTriggerIntegrityCheck() was removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944671138
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java:
##########
@@ -1346,4 +1349,56 @@ private TTLManager getTTLManager() {
private ExternalServiceManager getServiceManager() {
return configManager.getExternalServiceManager();
}
+
+ /**
+ * Check if all DataNodes are registered and running, then trigger integrity check. This method
+ * should be called after each DataNode registration.
+ */
+ private void checkAndTriggerIntegrityCheck() {
+ // Only trigger integrity check if this ConfigNode is the leader
+ if (!configManager.getConsensusManager().isLeader()) {
+ return;
+ }
+
+ // Get all registered DataNodes
+ List<TDataNodeConfiguration> registeredDataNodes = getRegisteredDataNodes();
+
+ // Check if all registered DataNodes are running
+ boolean allDataNodesRunning =
+ registeredDataNodes.stream()
+ .allMatch(
+ dataNode -> {
+ Integer dataNodeId = dataNode.getLocation().getDataNodeId();
+ NodeStatus status = getLoadManager().getLoadCache().getNodeStatus(dataNodeId);
+ return status == NodeStatus.Running;
+ });
+
+ if (allDataNodesRunning && !registeredDataNodes.isEmpty()) {
+ LOGGER.info(
+ "All {} DataNodes are registered and running, triggering data partition table integrity check",
+ registeredDataNodes.size());
+
+ // Trigger integrity check asynchronously
+ try {
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ LOGGER.info("Data partition table integrity check procedure submitted successfully");
Review Comment:
These codes have some problems, can't pass tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944661405
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java:
##########
@@ -352,6 +352,9 @@ public DataSet registerDataNode(TDataNodeRegisterReq req) {
// Adjust the maximum RegionGroup number of each Database
getClusterSchemaManager().adjustMaxRegionGroupNum();
+ // Check if all DataNodes are registered and trigger integrity check if needed
+ checkAndTriggerIntegrityCheck();
+
Review Comment:
The line was removed that It said checkAndTriggerIntegrityCheck()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957437635
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957319604
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeoutInMs());
+
+ while (true) {
+ List<Integer> dnList =
+ configManager
+ .getLoadManager()
+ .filterDataNodeThroughStatus(NodeStatus.Running);
+ if (dnList != null && !dnList.isEmpty()) {
+ LOGGER.info("Starting dataPartitionTableIntegrityCheck...");
+ TSStatus status =
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOGGER.error("Data partition table integrity check failed!");
Review Comment:
Changed yet
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeoutInMs());
+
+ while (true) {
+ List<Integer> dnList =
+ configManager
+ .getLoadManager()
+ .filterDataNodeThroughStatus(NodeStatus.Running);
+ if (dnList != null && !dnList.isEmpty()) {
+ LOGGER.info("Starting dataPartitionTableIntegrityCheck...");
+ TSStatus status =
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOGGER.error("Data partition table integrity check failed!");
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957309164
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeoutInMs());
+
+ while (true) {
+ List<Integer> dnList =
+ configManager
+ .getLoadManager()
+ .filterDataNodeThroughStatus(NodeStatus.Running);
+ if (dnList != null && !dnList.isEmpty()) {
+ LOGGER.info("Starting dataPartitionTableIntegrityCheck...");
+ TSStatus status =
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOGGER.error("Data partition table integrity check failed!");
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957386011
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+ // Set default value
+ resp.setDatabaseScopedDataPartitionTables(Collections.emptyList());
+ try {
+ currentGeneratorFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ parseGenerationStatus(resp);
+ if (currentGenerator.getStatus().equals(DataPartitionTableGenerator.TaskStatus.COMPLETED)) {
+ boolean success = false;
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = new ArrayList<>();
+ Map<String, DataPartitionTable> dataPartitionTableMap = currentGenerator.getDatabasePartitionTableMap();
+ if (!dataPartitionTableMap.isEmpty()) {
+ for (Map.Entry<String, DataPartitionTable> entry : dataPartitionTableMap.entrySet()) {
+ String database = entry.getKey();
+ DataPartitionTable dataPartitionTable = entry.getValue();
+ if (!StringUtils.isEmpty(database) && dataPartitionTable != null) {
+ DatabaseScopedDataPartitionTable databaseScopedDataPartitionTable = new DatabaseScopedDataPartitionTable(database, dataPartitionTable);
+ databaseScopedDataPartitionTableList.add(databaseScopedDataPartitionTable);
+ success = true;
+ }
+ }
+ }
+
+ if (success) {
+ List<ByteBuffer> result = serializeDatabaseScopedTableList(databaseScopedDataPartitionTableList);
+ resp.setDatabaseScopedDataPartitionTables(result);
+
+ // Clear current generator
+ currentGenerator = null;
+ }
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+ return resp;
+ }
+
+ private <T> void parseGenerationStatus(T resp) {
+ if (resp instanceof TGenerateDataPartitionTableResp) {
+ handleResponse((TGenerateDataPartitionTableResp) resp);
+ } else {
+ handleResponse((TGenerateDataPartitionTableHeartbeatResp) resp);
+ }
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableResp resp) {
+ updateResponse(resp);
+ }
+
+ private void handleResponse(TGenerateDataPartitionTableHeartbeatResp resp) {
+ updateResponse(resp);
+ }
+
+ private <T> void updateResponse(T resp) {
+ if (currentGenerator == null) return;
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957232784
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (localEarliestSlot.getStartTime() > TimePartitionUtils.getTimeByPartitionId(earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ }
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Flow requestPartitionTablesHeartBeat() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+
+ switch (state) {
+ case SUCCESS:
+ List<ByteBuffer> byteBufferList = resp.getDatabaseScopedDataPartitionTables();
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = deserializeDatabaseScopedTableList(byteBufferList);
+ dataPartitionTables.put(dataNodeId, databaseScopedDataPartitionTableList);
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ try {
+ Thread.sleep(HEART_BEAT_REQUEST_RATE);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.error("Error checking DataPartitionTable status due to thread interruption.");
+ }
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2958151938
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGeneratorFuture = currentGenerator.startGeneration();
+ parseGenerationStatus(resp);
Review Comment:
> Add some synchronization to avoid creating multiple generators if you do not intend to guarantee that only the CN leader will issue the repair.
Changed yet, see here
<img width="2682" height="1083" alt="image" src="https://github.com/user-attachments/assets/49ca3d01-87ad-4dc8-bbf0-f4923cfbd66b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 merged PR #17279:
URL: https://github.com/apache/iotdb/pull/17279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2954305445
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957260660
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (localEarliestSlot.getStartTime() > TimePartitionUtils.getTimeByPartitionId(earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ }
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Flow requestPartitionTablesHeartBeat() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+
+ switch (state) {
+ case SUCCESS:
+ List<ByteBuffer> byteBufferList = resp.getDatabaseScopedDataPartitionTables();
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = deserializeDatabaseScopedTableList(byteBufferList);
+ dataPartitionTables.put(dataNodeId, databaseScopedDataPartitionTableList);
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ try {
+ Thread.sleep(HEART_BEAT_REQUEST_RATE);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.error("Error checking DataPartitionTable status due to thread interruption.");
+ }
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+
+ dataPartitionTables.forEach((k, v) -> v.forEach(databaseScopedDataPartitionTable -> {
+ if (!databaseScopedDataPartitionTable.getDatabase().equals(database)) {
+ return;
+ }
+ finalDataPartitionTables.put(database, new DataPartitionTable(finalDataPartitionMap).merge(databaseScopedDataPartitionTable.getDataPartitionTable()));
+ }));
+ }
+
+ LOG.info("DataPartitionTables merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
+ try {
+ CreateDataPartitionPlan createPlan = new CreateDataPartitionPlan();
+ Map<String, DataPartitionTable> assignedDataPartition = new HashMap<>();
+ for (String database : lostDataPartitionsOfDatabases) {
+ assignedDataPartition.put(database, finalDataPartitionTables.get(database));
+ }
+ createPlan.setAssignedDataPartition(assignedDataPartition);
+ TSStatus tsStatus = env.getConfigManager().getConsensusManager().write(createPlan);
+
+ if (tsStatus.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.info("DataPartitionTable successfully written to raft log");
+ break;
+ } else {
+ LOG.error("Failed to write DataPartitionTable to raft log");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("Failed to write DataPartitionTable to raft log"));
+ }
+ } catch (Exception e) {
+ LOG.error("Error writing DataPartitionTable to raft log", e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ }
+ failedCnt++;
+ }
+
+ return getFlow();
+ }
+
+ /** Determine whether there are still DataNode nodes with failed execution of a certain step in this round. If such nodes exist, calculate the skipDataNodes and exclude these nodes when requesting the list of DataNode nodes in the cluster for the next round; if no such nodes exist, it means the procedure has been completed */
+ private Flow getFlow() {
+ if (!failedDataNodes.isEmpty()) {
+ allDataNodes.removeAll(failedDataNodes);
+ skipDataNodes = new HashSet<>(allDataNodes);
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ } else {
+ skipDataNodes.clear();
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ public void serialize(final DataOutputStream stream) throws IOException {
+ super.serialize(stream);
+
+ // Serialize earliestTimeslots
+ stream.writeInt(earliestTimeslots.size());
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ stream.writeUTF(entry.getKey());
+ stream.writeLong(entry.getValue());
+ }
+
+ // Serialize dataPartitionTables count
+ stream.writeInt(dataPartitionTables.size());
+ for (Map.Entry<Integer, List<DatabaseScopedDataPartitionTable>> entry : dataPartitionTables.entrySet()) {
+ stream.writeInt(entry.getKey());
+
+ List<DatabaseScopedDataPartitionTable> tableList = entry.getValue();
+ stream.writeInt(tableList.size());
+
+ for (DatabaseScopedDataPartitionTable table : tableList) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
+
+ TTransport transport = new TIOStreamTransport(dos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ table.serialize(dos, protocol);
+
+ byte[] data = baos.toByteArray();
+ // Length of data written for a single object
+ stream.writeInt(data.length);
+ // data written for a single object
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed for dataNodeId: {}", this.getClass().getSimpleName(), entry.getKey(), e);
+ throw new IOException("Failed to serialize dataPartitionTables", e);
+ }
+ }
+ }
+
+ stream.writeInt(lostDataPartitionsOfDatabases.size());
+ for (String database : lostDataPartitionsOfDatabases) {
+ stream.writeUTF(database);
+ }
+
+ if (finalDataPartitionTables != null && !finalDataPartitionTables.isEmpty()) {
+ stream.writeInt(finalDataPartitionTables.size());
+
+ for (Map.Entry<String, DataPartitionTable> entry : finalDataPartitionTables.entrySet()) {
+ stream.writeUTF(entry.getKey());
+
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
+
+ TTransport transport = new TIOStreamTransport(dos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ entry.getValue().serialize(dos, protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize finalDataPartitionTables failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize finalDataPartitionTables", e);
+ }
+ }
+ } else {
+ stream.writeInt(0);
+ }
+
+ stream.writeInt(skipDataNodes.size());
+ for (TDataNodeConfiguration skipDataNode : skipDataNodes) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
+ TTransport transport = new TIOStreamTransport(baos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ skipDataNode.write(protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (TException e) {
+ LOG.error("Failed to serialize skipDataNode", e);
+ throw new IOException("Failed to serialize skipDataNode", e);
+ }
+ }
+
+ stream.writeInt(failedDataNodes.size());
+ for (TDataNodeConfiguration failedDataNode : failedDataNodes) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
+ TTransport transport = new TIOStreamTransport(baos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ failedDataNode.write(protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (TException e) {
+ LOG.error("Failed to serialize failedDataNode", e);
+ throw new IOException("Failed to serialize failedDataNode", e);
+ }
+ }
+ }
+
+ @Override
+ public void deserialize(final ByteBuffer byteBuffer) {
+ super.deserialize(byteBuffer);
+
+ // Deserialize earliestTimeslots
+ int earliestTimeslotsSize = byteBuffer.getInt();
+ earliestTimeslots = new ConcurrentHashMap<>();
+ for (int i = 0; i < earliestTimeslotsSize; i++) {
+ String database = ReadWriteIOUtils.readString(byteBuffer);
+ long timeslot = byteBuffer.getLong();
+ earliestTimeslots.put(database, timeslot);
+ }
+
+ // Deserialize dataPartitionTables count
+ int dataPartitionTablesSize = byteBuffer.getInt();
+ dataPartitionTables = new ConcurrentHashMap<>();
+ for (int i = 0; i < dataPartitionTablesSize; i++) {
+ int dataNodeId = byteBuffer.getInt();
+ int listSize = byteBuffer.getInt();
+
+ List<DatabaseScopedDataPartitionTable> tableList = new ArrayList<>(listSize);
+
+ for (int j = 0; j < listSize; j++) {
+ int dataSize = byteBuffer.getInt();
+ byte[] bytes = new byte[dataSize];
+ byteBuffer.get(bytes);
+
+ try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bais)) {
+
+ TTransport transport = new TIOStreamTransport(dis);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ DatabaseScopedDataPartitionTable table =
+ DatabaseScopedDataPartitionTable.deserialize(dis, protocol);
+ tableList.add(table);
+
+ } catch (IOException | TException e) {
+ LOG.error("{} deserialize failed for dataNodeId: {}", this.getClass().getSimpleName(), dataNodeId, e);
+ throw new RuntimeException("Failed to deserialize dataPartitionTables", e);
+ }
+ }
+
+ dataPartitionTables.put(dataNodeId, tableList);
+ }
+
+ int lostDataPartitionsOfDatabasesSize = byteBuffer.getInt();
+ for (int i = 0; i < lostDataPartitionsOfDatabasesSize; i++) {
+ String database = ReadWriteIOUtils.readString(byteBuffer);
+ lostDataPartitionsOfDatabases.add(database);
+ }
+
+ // Deserialize finalDataPartitionTable size
+ int finalDataPartitionTablesSize = byteBuffer.getInt();
+ finalDataPartitionTables = new ConcurrentHashMap<>();
+
+ for (int i = 0; i < finalDataPartitionTablesSize; i++) {
+ String database = ReadWriteIOUtils.readString(byteBuffer);
+
+ int dataSize = byteBuffer.getInt();
+ byte[] bytes = new byte[dataSize];
+ byteBuffer.get(bytes);
+
+ try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
+ DataInputStream dis = new DataInputStream(bais)) {
+
+ TTransport transport = new TIOStreamTransport(dis);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(dis, protocol);
+
+ finalDataPartitionTables.put(database, dataPartitionTable);
+
+ } catch (IOException | TException e) {
+ LOG.error("{} deserialize finalDataPartitionTables failed", this.getClass().getSimpleName(), e);
+ throw new RuntimeException("Failed to deserialize finalDataPartitionTables", e);
+ }
+ }
+
+ skipDataNodes = new HashSet<>();
+ int skipDataNodesSize = byteBuffer.getInt();
+ for (int i = 0; i < skipDataNodesSize; i++) {
+ int size = byteBuffer.getInt();
+ byte[] bytes = new byte[size];
+ byteBuffer.get(bytes);
+
+ try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes)) {
+ TTransport transport = new TIOStreamTransport(bais);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ TDataNodeConfiguration dataNode = new TDataNodeConfiguration();
+ dataNode.read(protocol);
+ skipDataNodes.add(dataNode);
+ } catch (TException | IOException e) {
+ LOG.error("Failed to deserialize skipDataNode", e);
+ throw new RuntimeException(e);
+ }
+ }
+
+ failedDataNodes = new HashSet<>();
+ int failedDataNodesSize = byteBuffer.getInt();
+ for (int i = 0; i < failedDataNodesSize; i++) {
+ int size = byteBuffer.getInt();
+ byte[] bytes = new byte[size];
+ byteBuffer.get(bytes);
+
+ try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes)) {
+ TTransport transport = new TIOStreamTransport(bais);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+
+ TDataNodeConfiguration dataNode = new TDataNodeConfiguration();
+ dataNode.read(protocol);
+ failedDataNodes.add(dataNode);
+ } catch (TException | IOException e) {
+ LOG.error("Failed to deserialize failedDataNode", e);
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ private List<DatabaseScopedDataPartitionTable> deserializeDatabaseScopedTableList(List<ByteBuffer> dataList) {
+ if (dataList == null || dataList.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ List<DatabaseScopedDataPartitionTable> result = new ArrayList<>(dataList.size());
+
+ for (ByteBuffer data : dataList) {
+ if (data == null || data.remaining() == 0) {
+ LOG.warn("Skipping empty ByteBuffer during deserialization");
+ continue;
+ }
+
+ try {
+ ByteBuffer dataBuffer = data.duplicate();
+
+ DatabaseScopedDataPartitionTable table =
+ DatabaseScopedDataPartitionTable.deserialize(dataBuffer);
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957343305
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.commons.utils.rateLimiter;
+
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.LockSupport;
+
+/**
+ * A global leaky-bucket rate limiter for bytes throughput. Features: - Strict throughput limiting
+ * (no burst) - Smooth bandwidth shaping - Thread-safe - Fair for multi-thread - Low contention
+ */
+public class LeakyBucketRateLimiter {
Review Comment:
> May simply use com.google.common.util.concurrent.RateLimiter
Mainly I remember we hope to avoid too many external packages to carry into current repository. So I did it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957355112
##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -742,6 +742,21 @@ failure_detector_phi_acceptable_pause_in_ms=10000
# Datatype: double(percentage)
disk_space_warning_threshold=0.05
+# The number of threads used for parallel scanning in the partition table recovery
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_worker_num=10
+
+# Limit the number of bytes read per second from a file, the unit is MB
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_max_read_bytes_per_second=10
+
+# Set a timeout to wait for all datanodes complete startup, the unit is ms
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_wait_all_dn_up_timeout=60000
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944059689
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
Review Comment:
We already discussed this, no need to think about whether the request is synchronous.
I have split it to two steps so that don't be affraid the task have not been done when rpc request timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944123825
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.commons.utils.rateLimiter;
+
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.LockSupport;
+
+/**
+ * A global leaky-bucket rate limiter for bytes throughput. Features: - Strict throughput limiting
+ * (no burst) - Smooth bandwidth shaping - Thread-safe - Fair for multi-thread - Low contention
+ */
+public class LeakyBucketRateLimiter {
+ /** bytes per second */
+ private volatile long bytesPerSecond;
+
+ /** start time */
+ private final long startTimeNs;
+
+ /** total consumed bytes */
+ private final AtomicLong totalBytes = new AtomicLong(0);
+
+ public LeakyBucketRateLimiter(long bytesPerSecond) {
+ if (bytesPerSecond <= 0) {
+ throw new IllegalArgumentException("bytesPerSecond must be > 0");
+ }
+ this.bytesPerSecond = bytesPerSecond;
+ this.startTimeNs = System.nanoTime();
+ }
+
+ /**
+ * Acquire permission for reading bytes.
+ *
+ * <p>This method will block if reading too fast.
+ */
+ public void acquire(long bytes) {
+ if (bytes <= 0) {
+ return;
+ }
+
+ long currentTotal = totalBytes.addAndGet(bytes);
+
+ long expectedTimeNs = expectedTimeNs(currentTotal);
+ long now = System.nanoTime();
+
+ long sleepNs = expectedTimeNs - now;
+
+ if (sleepNs > 0) {
+ LockSupport.parkNanos(sleepNs);
+ }
+ }
+
+ /**
+ * Try acquire without blocking.
+ *
+ * @return true if allowed immediately
+ */
+ public boolean tryAcquire(long bytes) {
+ if (bytes <= 0) {
+ return true;
+ }
+
+ long currentTotal = totalBytes.addAndGet(bytes);
+
+ long expectedTimeNs = expectedTimeNs(currentTotal);
+ long now = System.nanoTime();
+
+ if (expectedTimeNs <= now) {
+ return true;
+ }
+
+ // rollback
+ totalBytes.addAndGet(-bytes);
+ return false;
+ }
+
+ /** Update rate dynamically. */
+ public void setRate(long newBytesPerSecond) {
+ if (newBytesPerSecond <= 0) {
+ throw new IllegalArgumentException("bytesPerSecond must be > 0");
+ }
+ this.bytesPerSecond = newBytesPerSecond;
+ }
+
+ /** Current rate. */
+ public long getRate() {
+ return bytesPerSecond;
+ }
+
+ /** Total bytes processed. */
+ public long getTotalBytes() {
+ return totalBytes.get();
+ }
+
+ /** Expected time based on bytes processed. */
+ private long expectedTimeNs(long totalBytes) {
+ return startTimeNs + (totalBytes * 1_000_000_000L) / bytesPerSecond;
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944156669
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/timeindex/FileTimeIndex.java:
##########
@@ -120,6 +121,41 @@ public Set<IDeviceID> getDevices(String tsFilePath, TsFileResource tsFileResourc
}
}
+ @Override
+ public Set<IDeviceID> getDevices(
+ String tsFilePath, TsFileResource tsFileResource, LeakyBucketRateLimiter limiter) {
+ tsFileResource.readLock();
+ try (InputStream inputStream =
+ FSFactoryProducer.getFSFactory()
+ .getBufferedInputStream(tsFilePath + TsFileResource.RESOURCE_SUFFIX)) {
+ // The first byte is VERSION_NUMBER, second byte is timeIndexType.
+ byte[] bytes = ReadWriteIOUtils.readBytes(inputStream, 2);
+ limiter.acquire(bytes.length);
+ if (bytes[1] == ARRAY_DEVICE_TIME_INDEX_TYPE) {
+ return ArrayDeviceTimeIndex.getDevices(inputStream);
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944157906
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/partition/SeriesPartitionTable.java:
##########
@@ -73,7 +73,11 @@ public Map<TTimePartitionSlot, List<TConsensusGroupId>> getSeriesPartitionMap()
}
public void putDataPartition(TTimePartitionSlot timePartitionSlot, TConsensusGroupId groupId) {
- seriesPartitionMap.computeIfAbsent(timePartitionSlot, empty -> new Vector<>()).add(groupId);
+ seriesPartitionMap.computeIfAbsent(timePartitionSlot, empty -> new Vector<>());
+ List<TConsensusGroupId> groupList = seriesPartitionMap.get(timePartitionSlot);
+ if (!groupList.contains(groupId)) {
+ groupList.add(groupId);
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2954316298
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944866481
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
Review Comment:
> Change this map into a local variable.
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957309164
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "[DataPartitionIntegrity] Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeoutInMs());
+
+ while (true) {
+ List<Integer> dnList =
+ configManager
+ .getLoadManager()
+ .filterDataNodeThroughStatus(NodeStatus.Running);
+ if (dnList != null && !dnList.isEmpty()) {
+ LOGGER.info("Starting dataPartitionTableIntegrityCheck...");
+ TSStatus status =
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ if (status.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOGGER.error("Data partition table integrity check failed!");
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957323398
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (localEarliestSlot.getStartTime() > TimePartitionUtils.getTimeByPartitionId(earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ }
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Flow requestPartitionTablesHeartBeat() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+
+ switch (state) {
+ case SUCCESS:
+ List<ByteBuffer> byteBufferList = resp.getDatabaseScopedDataPartitionTables();
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = deserializeDatabaseScopedTableList(byteBufferList);
+ dataPartitionTables.put(dataNodeId, databaseScopedDataPartitionTableList);
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ try {
+ Thread.sleep(HEART_BEAT_REQUEST_RATE);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.error("Error checking DataPartitionTable status due to thread interruption.");
+ }
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957322385
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "jt2594838 (via GitHub)" <gi...@apache.org>.
jt2594838 commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2963753137
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3392,71 +3366,62 @@ private long findEarliestTimeslotInDatabase(File databaseDir, Map<String, Long>
ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
new ThreadPoolExecutor.CallerRunsPolicy());
- try (Stream<Path> databasePaths = Files.list(databaseDir.toPath())) {
- databasePaths
- .filter(Files::isDirectory)
- .forEach(
- regionPath -> {
- Future<?> future =
- findEarliestTimeSlotExecutor.submit(
- () -> {
- try (Stream<Path> regionPaths = Files.list(regionPath)) {
- regionPaths
- .filter(Files::isDirectory)
- .forEach(
- timeSlotPath -> {
- try {
- Optional<Path> matchedFile =
- Files.find(
- timeSlotPath,
- 1,
- (path, attrs) ->
- attrs.isRegularFile()
- && path.toString()
- .endsWith(
- DataPartitionTableGenerator
- .SCAN_FILE_SUFFIX_NAME))
- .findFirst();
- if (!matchedFile.isPresent()) {
- return;
- }
- String timeSlotName = timeSlotPath.getFileName().toString();
- long timeslot = Long.parseLong(timeSlotName);
- databaseEarliestRegionMap.compute(
- databaseName,
- (k, v) ->
- v == null ? timeslot : Math.min(v, timeslot));
- } catch (IOException e) {
- LOGGER.error(
- "Failed to find any {} files in the {} directory",
- DataPartitionTableGenerator.SCAN_FILE_SUFFIX_NAME,
- timeSlotPath,
- e);
- }
- });
- } catch (IOException e) {
- LOGGER.error("Failed to scan {}", regionPath, e);
- }
- });
- futureList.add(future);
- });
- } catch (IOException e) {
- LOGGER.error("Failed to walk database directory: {}", databaseDir, e);
- }
-
- for (Future<?> future : futureList) {
- try {
- future.get();
- } catch (InterruptedException | ExecutionException e) {
- LOGGER.error("Failed to wait for task completion", e);
- Thread.currentThread().interrupt();
- }
+ for (DataRegion dataRegion : StorageEngine.getInstance().getAllDataRegions()) {
+ CompletableFuture<Void> regionFuture =
+ CompletableFuture.runAsync(
+ () -> {
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ String databaseName = dataRegion.getDatabaseName();
+ if (ignoreDatabase.contains(databaseName)) {
+ return;
+ }
+
+ tsFileManager.readLock();
+ List<TsFileResource> seqTsFileList = tsFileManager.getTsFileList(true);
+ List<TsFileResource> unseqTsFileList = tsFileManager.getTsFileList(false);
+ tsFileManager.readUnlock();
+
+ long earliestTimeSlotId = Long.MIN_VALUE;
+
+ earliestTimeSlotId = findEarliestTimeslotInFiles(seqTsFileList, earliestTimeSlotId);
+ earliestTimeSlotId =
+ findEarliestTimeslotInFiles(unseqTsFileList, earliestTimeSlotId);
+
+ if (earliestTimeSlotId == Long.MIN_VALUE) {
+ LOGGER.info("No time slot info is found in the seq and unseq directory");
+ return;
+ }
+
+ long finalEarliestTimeSlotId = earliestTimeSlotId;
+ earliestTimeslots.compute(
+ databaseName,
+ (k, v) ->
+ v == null ? finalEarliestTimeSlotId : Math.min(finalEarliestTimeSlotId, v));
+ },
Review Comment:
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager#getTimePartitions is enough
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -213,15 +219,17 @@ private void constructDataPartitionMap(
TSeriesPartitionSlot seriesSlotId =
seriesPartitionExecutor.getSeriesPartitionSlot(deviceId);
TTimePartitionSlot timePartitionSlot =
- new TTimePartitionSlot(TimePartitionUtils.getTimeByPartitionId(timeSlotId));
+ new TTimePartitionSlot(TimePartitionUtils.getStartTimeByPartitionId(timeSlotId));
dataPartitionMap
.computeIfAbsent(
seriesSlotId, empty -> newSeriesPartitionTable(consensusGroupId, timeSlotId))
.putDataPartition(timePartitionSlot, consensusGroupId);
}
processedFiles.incrementAndGet();
+ totalFiles.incrementAndGet();
} catch (Exception e) {
failedFiles.incrementAndGet();
+ totalFiles.incrementAndGet();
Review Comment:
If you are using the totalFiles to calculate the progress, then this is not the right way to update it.
You should either calculate it from the beginning or use another method for progress calculation.
For example, the number of total regions/partitions and the number processed.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -658,42 +718,49 @@ public void serialize(final DataOutputStream stream) throws IOException {
// data written for a single object
stream.write(data);
} catch (IOException | TException e) {
- LOG.error("{} serialize failed for dataNodeId: {}", this.getClass().getSimpleName(), entry.getKey(), e);
+ LOG.error(
+ "[DataPartitionIntegrity] {} serialize failed for dataNodeId: {}",
+ this.getClass().getSimpleName(),
+ entry.getKey(),
+ e);
throw new IOException("Failed to serialize dataPartitionTables", e);
}
}
}
- stream.writeInt(lostDataPartitionsOfDatabases.size());
- for (String database : lostDataPartitionsOfDatabases) {
- stream.writeUTF(database);
+ stream.writeInt(databasesWithLostDataPartition.size());
+ for (String database : databasesWithLostDataPartition) {
+ ReadWriteIOUtils.write(database, stream);
}
- if (finalDataPartitionTables != null && !finalDataPartitionTables.isEmpty()) {
- stream.writeInt(finalDataPartitionTables.size());
+ if (finalDataPartitionTables != null && !finalDataPartitionTables.isEmpty()) {
+ stream.writeInt(finalDataPartitionTables.size());
- for (Map.Entry<String, DataPartitionTable> entry : finalDataPartitionTables.entrySet()) {
- stream.writeUTF(entry.getKey());
+ for (Map.Entry<String, DataPartitionTable> entry : finalDataPartitionTables.entrySet()) {
+ ReadWriteIOUtils.write(entry.getKey(), stream);
- try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
- DataOutputStream dos = new DataOutputStream(baos)) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream dos = new DataOutputStream(baos)) {
- TTransport transport = new TIOStreamTransport(dos);
- TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ TTransport transport = new TIOStreamTransport(dos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
- entry.getValue().serialize(dos, protocol);
+ entry.getValue().serialize(dos, protocol);
- byte[] data = baos.toByteArray();
- stream.writeInt(data.length);
- stream.write(data);
- } catch (IOException | TException e) {
- LOG.error("{} serialize finalDataPartitionTables failed", this.getClass().getSimpleName(), e);
- throw new IOException("Failed to serialize finalDataPartitionTables", e);
- }
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
Review Comment:
May use PublicBAOS to replace baos.toByteArray() with baos.getBuf()
##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -742,6 +742,21 @@ failure_detector_phi_acceptable_pause_in_ms=10000
# Datatype: double(percentage)
disk_space_warning_threshold=0.05
+# The number of threads used for parallel scanning in the partition table recovery
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_worker_num=10
+
+# Limit the number of bytes read per second from a file, the unit is MB
Review Comment:
for data partition repair
Add in the comment, not just explain it
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.commons.utils.rateLimiter;
+
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.LockSupport;
+
+/**
+ * A global leaky-bucket rate limiter for bytes throughput. Features: - Strict throughput limiting
+ * (no burst) - Smooth bandwidth shaping - Thread-safe - Fair for multi-thread - Low contention
+ */
+public class LeakyBucketRateLimiter {
Review Comment:
We have already used it in other modules, so using it here will not add a new dependency
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3150,23 +3147,13 @@ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
try {
- Map<String, Long> earliestTimeslots = new HashMap<>();
-
- // Get data directories from configuration
- String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
-
- for (String dataDir : dataDirs) {
- File dir = new File(dataDir);
- if (dir.exists() && dir.isDirectory()) {
- processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
- }
- }
+ Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+ processDataDirectoryForEarliestTimeslots(earliestTimeslots);
Review Comment:
No longer DataDirectory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957440557
##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -742,6 +742,21 @@ failure_detector_phi_acceptable_pause_in_ms=10000
# Datatype: double(percentage)
disk_space_warning_threshold=0.05
+# The number of threads used for parallel scanning in the partition table recovery
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_worker_num=10
+
+# Limit the number of bytes read per second from a file, the unit is MB
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_max_read_bytes_per_second=10
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2929351028
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/enums/DataPartitionTableGeneratorState.java:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.commons.enums;
+
+public enum DataPartitionTableGeneratorState {
+ SUCCESS(0),
+ FAILED(1),
+ IN_PROGRESS(2),
+ UNKNOWN(-1);
+
+ private final int code;
+
+ DataPartitionTableGeneratorState(int code) {
+ this.code = code;
+ }
+
+ public int getCode() {
+ return code;
+ }
+
+ /**
+ * get DataNodeRemoveState by code
+ *
+ * @param code code
+ * @return DataNodeRemoveState
Review Comment:
The Javadoc for `getStateByCode` incorrectly references `DataNodeRemoveState` in the description and `@return`. Update it to reference `DataPartitionTableGeneratorState` to avoid confusion.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+ byte[] empty = new byte[0];
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGenerator.startGeneration().get(timeoutMs, TimeUnit.MILLISECONDS);
+
+ if (currentGenerator != null) {
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage("DataPartitionTable generation interrupted");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ DataPartitionTable dataPartitionTable = currentGenerator.getDataPartitionTable();
+ if (dataPartitionTable != null) {
+ byte[] result = serializeDataPartitionTable(dataPartitionTable);
+ resp.setDataPartitionTable(result);
+ }
+
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ default:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ }
+
+ // Clear current generator
+ currentGenerator = null;
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+
+ try {
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ DataPartitionTableGenerator.TaskStatus status = currentGenerator.getStatus();
+
+ switch (status) {
+ case IN_PROGRESS:
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage(
+ String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100));
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case FAILED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ default:
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("Unknown task status: " + status);
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ /** Process data directory to find the earliest timeslots for each database. */
+ private void processDataDirectoryForEarliestTimeslots(
+ File dataDir, Map<String, Long> earliestTimeslots) {
+ try {
+ Files.list(dataDir.toPath())
+ .filter(Files::isDirectory)
+ .forEach(
+ sequenceTypePath -> {
+ try {
+ Files.list(sequenceTypePath)
+ .filter(Files::isDirectory)
+ .forEach(
+ dbPath -> {
+ String databaseName = dbPath.getFileName().toString();
+ if (DataPartitionTableGenerator.IGNORE_DATABASE.contains(
+ databaseName)) {
+ return;
+ }
+ databaseEarliestRegionMap.computeIfAbsent(
+ databaseName, key -> Long.MAX_VALUE);
+ long earliestTimeslot = findEarliestTimeslotInDatabase(dbPath.toFile());
+
+ if (earliestTimeslot != Long.MAX_VALUE) {
+ earliestTimeslots.merge(databaseName, earliestTimeslot, Math::min);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error(
+ "Failed to process data directory: {}", sequenceTypePath.toFile(), e);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to process data directory: {}", dataDir, e);
+ }
+ }
+
+ /** Find the earliest timeslot in a database directory. */
+ private long findEarliestTimeslotInDatabase(File databaseDir) {
+ String databaseName = databaseDir.getName();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ try {
+ Files.list(databaseDir.toPath())
+ .filter(Files::isDirectory)
+ .forEach(
+ regionPath -> {
+ Future<?> future =
+ findEarliestTimeSlotExecutor.submit(
+ () -> {
+ try {
+ Files.list(regionPath)
+ .filter(Files::isDirectory)
+ .forEach(
+ timeSlotPath -> {
+ try {
+ Optional<Path> matchedFile =
+ Files.find(
+ timeSlotPath,
+ 1,
+ (path, attrs) ->
+ attrs.isRegularFile()
+ && path.toString()
+ .endsWith(
+ DataPartitionTableGenerator
+ .SCAN_FILE_SUFFIX_NAME))
+ .findFirst();
+ if (!matchedFile.isPresent()) {
+ return;
+ }
+ String timeSlotName = timeSlotPath.getFileName().toString();
+ long timeslot = Long.parseLong(timeSlotName);
+ if (timeslot
+ < databaseEarliestRegionMap.get(databaseName)) {
+ databaseEarliestRegionMap.put(databaseName, timeslot);
+ }
Review Comment:
The parallel scan updates `databaseEarliestRegionMap` with a non-atomic check-then-put, which can race and lose the true minimum. Use `compute`/`merge` with `Math::min` so updates are atomic under concurrency.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
+
Review Comment:
`databaseEarliestRegionMap` is a mutable field used to accumulate the per-database minimum timeslot, but it is never cleared/reset at the start of `getEarliestTimeslots()`. That can cause stale minima to leak across RPC calls (e.g., after data cleanup). Prefer making this map local to the request, or explicitly clear/reset it before scanning.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/partition/SeriesPartitionTable.java:
##########
@@ -73,7 +73,11 @@ public Map<TTimePartitionSlot, List<TConsensusGroupId>> getSeriesPartitionMap()
}
public void putDataPartition(TTimePartitionSlot timePartitionSlot, TConsensusGroupId groupId) {
- seriesPartitionMap.computeIfAbsent(timePartitionSlot, empty -> new Vector<>()).add(groupId);
+ seriesPartitionMap.computeIfAbsent(timePartitionSlot, empty -> new Vector<>());
+ List<TConsensusGroupId> groupList = seriesPartitionMap.get(timePartitionSlot);
+ if (!groupList.contains(groupId)) {
+ groupList.add(groupId);
Review Comment:
`putDataPartition` uses a `contains` + `add` sequence to avoid duplicates, but this check-then-act is not atomic even though the table is used concurrently. Under concurrent calls, duplicates can still be inserted. Consider synchronizing on the list or using a Set-based structure if uniqueness is required.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+ byte[] empty = new byte[0];
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGenerator.startGeneration().get(timeoutMs, TimeUnit.MILLISECONDS);
+
+ if (currentGenerator != null) {
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage("DataPartitionTable generation interrupted");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ DataPartitionTable dataPartitionTable = currentGenerator.getDataPartitionTable();
+ if (dataPartitionTable != null) {
+ byte[] result = serializeDataPartitionTable(dataPartitionTable);
+ resp.setDataPartitionTable(result);
+ }
+
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ default:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ }
+
+ // Clear current generator
+ currentGenerator = null;
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+
+ try {
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ DataPartitionTableGenerator.TaskStatus status = currentGenerator.getStatus();
+
+ switch (status) {
+ case IN_PROGRESS:
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage(
+ String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100));
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case FAILED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ default:
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("Unknown task status: " + status);
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ /** Process data directory to find the earliest timeslots for each database. */
+ private void processDataDirectoryForEarliestTimeslots(
+ File dataDir, Map<String, Long> earliestTimeslots) {
+ try {
+ Files.list(dataDir.toPath())
+ .filter(Files::isDirectory)
+ .forEach(
+ sequenceTypePath -> {
+ try {
+ Files.list(sequenceTypePath)
+ .filter(Files::isDirectory)
+ .forEach(
+ dbPath -> {
+ String databaseName = dbPath.getFileName().toString();
+ if (DataPartitionTableGenerator.IGNORE_DATABASE.contains(
+ databaseName)) {
+ return;
+ }
+ databaseEarliestRegionMap.computeIfAbsent(
+ databaseName, key -> Long.MAX_VALUE);
+ long earliestTimeslot = findEarliestTimeslotInDatabase(dbPath.toFile());
+
+ if (earliestTimeslot != Long.MAX_VALUE) {
+ earliestTimeslots.merge(databaseName, earliestTimeslot, Math::min);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error(
+ "Failed to process data directory: {}", sequenceTypePath.toFile(), e);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to process data directory: {}", dataDir, e);
+ }
+ }
+
+ /** Find the earliest timeslot in a database directory. */
+ private long findEarliestTimeslotInDatabase(File databaseDir) {
+ String databaseName = databaseDir.getName();
+ List<Future<?>> futureList = new ArrayList<>();
+
+ try {
+ Files.list(databaseDir.toPath())
+ .filter(Files::isDirectory)
+ .forEach(
+ regionPath -> {
+ Future<?> future =
+ findEarliestTimeSlotExecutor.submit(
+ () -> {
+ try {
+ Files.list(regionPath)
+ .filter(Files::isDirectory)
+ .forEach(
+ timeSlotPath -> {
+ try {
+ Optional<Path> matchedFile =
+ Files.find(
+ timeSlotPath,
+ 1,
+ (path, attrs) ->
+ attrs.isRegularFile()
+ && path.toString()
+ .endsWith(
+ DataPartitionTableGenerator
+ .SCAN_FILE_SUFFIX_NAME))
+ .findFirst();
+ if (!matchedFile.isPresent()) {
+ return;
+ }
+ String timeSlotName = timeSlotPath.getFileName().toString();
+ long timeslot = Long.parseLong(timeSlotName);
+ if (timeslot
+ < databaseEarliestRegionMap.get(databaseName)) {
+ databaseEarliestRegionMap.put(databaseName, timeslot);
+ }
+ } catch (IOException e) {
+ LOGGER.error(
+ "Failed to find any {} files in the {} directory",
+ DataPartitionTableGenerator.SCAN_FILE_SUFFIX_NAME,
+ timeSlotPath,
+ e);
+ }
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to scan {}", regionPath, e);
+ }
+ });
+ futureList.add(future);
+ });
+ } catch (IOException e) {
+ LOGGER.error("Failed to walk database directory: {}", databaseDir, e);
+ }
+
+ for (Future<?> future : futureList) {
+ try {
+ future.get();
+ } catch (InterruptedException | ExecutionException e) {
+ LOGGER.error("Failed to wait for task completion", e);
+ Thread.currentThread().interrupt();
+ }
+ }
+ return databaseEarliestRegionMap.get(databaseName);
+ }
+
+ /** Serialize DataPartitionTable to ByteBuffer for RPC transmission. */
+ private byte[] serializeDataPartitionTable(DataPartitionTable dataPartitionTable) {
+ try (PublicBAOS baos = new PublicBAOS();
+ DataOutputStream oos = new DataOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ dataPartitionTable.serialize(oos, protocol);
+ return baos.getBuf();
Review Comment:
`serializeDataPartitionTable` returns `baos.getBuf()`, which includes the full internal buffer capacity (with trailing unused bytes). Return only the written region (e.g., copy up to `baos.size()`), otherwise deserialization can fail or read garbage.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+ }
+
+ if (finalDataPartitionMap.isEmpty()) {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (dnSeriesPartitionSlot == null || dnSeriesPartitionTable == null) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ dnSeriesPartitionSlot, k -> dnSeriesPartitionTable);
+ });
+ });
+ } else {
+ finalDataPartitionMap.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTable) -> {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (!tSeriesPartitionSlot.equals(dnSeriesPartitionSlot)) {
+ return;
+ }
+
+ if (seriesPartitionTable == null
+ || seriesPartitionTable.getSeriesPartitionMap() == null
+ || seriesPartitionTable.getSeriesPartitionMap().isEmpty()) {
+ finalDataPartitionMap.put(
+ tSeriesPartitionSlot, dnSeriesPartitionTable);
+ }
+
+ // dnDataPartitionTable merged to seriesPartitionTable
+ dnSeriesPartitionTable
+ .getSeriesPartitionMap()
+ .forEach(
+ (k, v) ->
+ v.forEach(
+ tConsensusGroupId -> {
+ if (seriesPartitionTable == null) {
+ return;
+ }
+ seriesPartitionTable.putDataPartition(
+ k, tConsensusGroupId);
+ }));
+ });
+ });
+ });
+ }
+
+ finalDataPartitionTable = new DataPartitionTable(finalDataPartitionMap);
+
+ LOG.info("DataPartitionTable merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTable == null) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
+ try {
+ CreateDataPartitionPlan createPlan = new CreateDataPartitionPlan();
+ Map<String, DataPartitionTable> assignedDataPartition = new HashMap<>();
+ assignedDataPartition.put(
+ lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);
Review Comment:
`writePartitionTableToRaft()` writes the merged `finalDataPartitionTable` under only the first database in `lostDataPartitionsOfDatabases`. If multiple databases need repair, the others are silently ignored, and the merged table (built across databases) is also not scoped per-database. Build and write an assigned DataPartitionTable per database, and write them all in the plan.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
Review Comment:
The recovery thread pool size is used as the `ArrayBlockingQueue` capacity. If the config is set to 0 (or negative), this will throw during service initialization. Validate `partitionTableRecoverWorkerNum` (min 1) when loading config or before constructing these executors.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/TimePartitionUtils.java:
##########
@@ -122,6 +130,23 @@ public static long getTimePartitionIdWithoutOverflow(long time) {
return partitionId.longValue();
}
+ public static long getTimeWithoutOverflow(long partitionId) {
+ BigInteger bigTime = bigTimePartitionInterval.multiply(BigInteger.valueOf(partitionId));
+ if (bigTime.compareTo(BigInteger.ZERO) > 0
+ || bigTime.remainder(bigTimePartitionInterval).equals(BigInteger.ZERO)) {
+ return bigTime.add(bigTimePartitionOrigin).longValue();
+ }
+ return BigInteger.valueOf(partitionId)
+ .add(BigInteger.ONE)
+ .multiply(bigTimePartitionInterval)
Review Comment:
In `getTimeWithoutOverflow`, `bigTime` is calculated as `interval * partitionId`, so `bigTime.remainder(interval)` is always 0. The condition is therefore always true and the `else` branch is dead code, making the method misleading; simplify it or document the intended edge case.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/timeindex/FileTimeIndex.java:
##########
@@ -120,6 +121,41 @@ public Set<IDeviceID> getDevices(String tsFilePath, TsFileResource tsFileResourc
}
}
+ @Override
+ public Set<IDeviceID> getDevices(
+ String tsFilePath, TsFileResource tsFileResource, LeakyBucketRateLimiter limiter) {
+ tsFileResource.readLock();
+ try (InputStream inputStream =
+ FSFactoryProducer.getFSFactory()
+ .getBufferedInputStream(tsFilePath + TsFileResource.RESOURCE_SUFFIX)) {
+ // The first byte is VERSION_NUMBER, second byte is timeIndexType.
+ byte[] bytes = ReadWriteIOUtils.readBytes(inputStream, 2);
+ limiter.acquire(bytes.length);
+ if (bytes[1] == ARRAY_DEVICE_TIME_INDEX_TYPE) {
+ return ArrayDeviceTimeIndex.getDevices(inputStream);
Review Comment:
The rate limiter is only charged for the first 2 bytes (`readBytes(..., 2)`), but the bulk of the time-index payload is read inside `ArrayDeviceTimeIndex.getDevices(inputStream)` / `PlainDeviceTimeIndex.getDevices(inputStream)` without limiting. If the goal is to cap disk throughput, apply limiting to all bytes read (e.g., via a counting/wrapping InputStream).
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
Review Comment:
This `forEach` marks a database as “lost” for every existing time slot that does not satisfy the earliest-timeslot condition, which will incorrectly flag most non-empty databases. The logic should instead detect loss only if *no* partition matches the earliest timeslot (i.e., check existence and short-circuit once found).
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
Review Comment:
`skipDataNodes`, `failedDataNodes`, and `heartBeatExecutor` are declared `static`, so concurrent procedure runs (or even retries) will share state across instances and the state will not be persisted correctly by procedure serialization. These should be instance fields (and ideally thread-safe collections) so each procedure execution is isolated and recoverable.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/ProcedureManager.java:
##########
@@ -1374,6 +1375,16 @@ public TSStatus createRegionGroups(
}
}
+ /** Used to repair the lost data partition table */
+ public TSStatus dataPartitionTableIntegrityCheck() {
+ DataPartitionTableIntegrityCheckProcedure procedure;
+ synchronized (this) {
+ procedure = new DataPartitionTableIntegrityCheckProcedure();
+ executor.submitProcedure(procedure);
+ }
+ return waitingProcedureFinished(procedure);
Review Comment:
`dataPartitionTableIntegrityCheck()` blocks by calling `waitingProcedureFinished(procedure)`. This makes callers (e.g., registration/startup hooks) stall while the scan runs. Consider providing a submit-only API (non-blocking) and using that for automatic triggers.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java:
##########
@@ -352,6 +352,9 @@ public DataSet registerDataNode(TDataNodeRegisterReq req) {
// Adjust the maximum RegionGroup number of each Database
getClusterSchemaManager().adjustMaxRegionGroupNum();
+ // Check if all DataNodes are registered and trigger integrity check if needed
+ checkAndTriggerIntegrityCheck();
+
Review Comment:
`checkAndTriggerIntegrityCheck()` is invoked synchronously during `registerDataNode()`. Since it calls `ProcedureManager.dataPartitionTableIntegrityCheck()` (which waits for procedure completion), DataNode registration can be blocked for a long time (or indefinitely), impacting cluster availability. Submit the procedure asynchronously (or return immediately) and add a guard to prevent repeated triggers on every registration.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
+
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ resp.setDatabaseToEarliestTimeslot(earliestTimeslots);
+
+ LOGGER.info("Retrieved earliest timeslots for {} databases", earliestTimeslots.size());
+
+ } catch (Exception e) {
+ LOGGER.error("Failed to get earliest timeslots", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GET_EARLIEST_TIMESLOTS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableResp generateDataPartitionTable(
+ TGenerateDataPartitionTableReq req) {
+ TGenerateDataPartitionTableResp resp = new TGenerateDataPartitionTableResp();
+ byte[] empty = new byte[0];
+
+ try {
+ // Check if there's already a task in the progress
+ if (currentGenerator != null
+ && currentGenerator.getStatus() == DataPartitionTableGenerator.TaskStatus.IN_PROGRESS) {
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage("DataPartitionTable generation is already in the progress");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ // Create generator for all data directories
+ int seriesSlotNum = IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionSlotNum();
+ String seriesPartitionExecutorClass =
+ IoTDBDescriptor.getInstance().getConfig().getSeriesPartitionExecutorClass();
+
+ currentGenerator =
+ new DataPartitionTableGenerator(
+ partitionTableRecoverExecutor,
+ req.getDatabases(),
+ seriesSlotNum,
+ seriesPartitionExecutorClass);
+ currentTaskId = System.currentTimeMillis();
+
+ // Start generation synchronously for now to return the data partition table immediately
+ currentGenerator.startGeneration().get(timeoutMs, TimeUnit.MILLISECONDS);
+
+ if (currentGenerator != null) {
+ switch (currentGenerator.getStatus()) {
+ case IN_PROGRESS:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage("DataPartitionTable generation interrupted");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ DataPartitionTable dataPartitionTable = currentGenerator.getDataPartitionTable();
+ if (dataPartitionTable != null) {
+ byte[] result = serializeDataPartitionTable(dataPartitionTable);
+ resp.setDataPartitionTable(result);
+ }
+
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ LOGGER.info("DataPartitionTable generation completed with task ID: {}", currentTaskId);
+ break;
+ default:
+ resp.setDataPartitionTable(empty);
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ }
+
+ // Clear current generator
+ currentGenerator = null;
+ } catch (Exception e) {
+ LOGGER.error("Failed to generate DataPartitionTable", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.GENERATE_DATA_PARTITION_TABLE,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ @Override
+ public TGenerateDataPartitionTableHeartbeatResp generateDataPartitionTableHeartbeat() {
+ TGenerateDataPartitionTableHeartbeatResp resp = new TGenerateDataPartitionTableHeartbeatResp();
+
+ try {
+ if (currentGenerator == null) {
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("No DataPartitionTable generation task found");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ return resp;
+ }
+
+ DataPartitionTableGenerator.TaskStatus status = currentGenerator.getStatus();
+
+ switch (status) {
+ case IN_PROGRESS:
+ resp.setErrorCode(DataPartitionTableGeneratorState.IN_PROGRESS.getCode());
+ resp.setMessage(
+ String.format(
+ "DataPartitionTable generation in progress: %.1f%%",
+ currentGenerator.getProgress() * 100));
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ case COMPLETED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.SUCCESS.getCode());
+ resp.setMessage("DataPartitionTable generation completed successfully");
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS));
+ break;
+ case FAILED:
+ resp.setErrorCode(DataPartitionTableGeneratorState.FAILED.getCode());
+ resp.setMessage(
+ "DataPartitionTable generation failed: " + currentGenerator.getErrorMessage());
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ default:
+ resp.setErrorCode(DataPartitionTableGeneratorState.UNKNOWN.getCode());
+ resp.setMessage("Unknown task status: " + status);
+ resp.setStatus(RpcUtils.getStatus(TSStatusCode.INTERNAL_SERVER_ERROR));
+ break;
+ }
+ } catch (Exception e) {
+ LOGGER.error("Failed to check DataPartitionTable generation status", e);
+ resp.setStatus(
+ onIoTDBException(
+ e,
+ OperationType.CHECK_DATA_PARTITION_TABLE_STATUS,
+ TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode()));
+ }
+
+ return resp;
+ }
+
+ /** Process data directory to find the earliest timeslots for each database. */
+ private void processDataDirectoryForEarliestTimeslots(
+ File dataDir, Map<String, Long> earliestTimeslots) {
+ try {
+ Files.list(dataDir.toPath())
+ .filter(Files::isDirectory)
+ .forEach(
+ sequenceTypePath -> {
+ try {
+ Files.list(sequenceTypePath)
Review Comment:
`Files.list(...)` returns a Stream that must be closed; this `Files.list(dataDir.toPath())` usage is not wrapped in try-with-resources and can leak file descriptors during scans. Wrap each `Files.list`/`Files.find`/`Files.walk` stream in try-with-resources.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupType;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.partition.executor.SeriesPartitionExecutor;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.commons.utils.rateLimiter.LeakyBucketRateLimiter;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Generator for DataPartitionTable by scanning tsfile resources. This class scans the data
+ * directory structure and builds a complete DataPartitionTable based on existing tsfiles.
+ */
+public class DataPartitionTableGenerator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DataPartitionTableGenerator.class);
+
+ // Task status
+ private volatile TaskStatus status = TaskStatus.NOT_STARTED;
+ private volatile String errorMessage;
+ private volatile DataPartitionTable dataPartitionTable;
+
+ // Progress tracking
+ private final AtomicInteger processedFiles = new AtomicInteger(0);
+ private final AtomicInteger failedFiles = new AtomicInteger(0);
+ private final AtomicLong totalFiles = new AtomicLong(0);
+
+ // Configuration
+ private String[] dataDirectories;
+ private final ExecutorService executor;
+ private final Set<String> databases;
+ private final int seriesSlotNum;
+ private final String seriesPartitionExecutorClass;
+
+ private static final int EXECUTOR_MAX_TIMEOUT = 60;
+
+ private static final LeakyBucketRateLimiter limiter =
+ new LeakyBucketRateLimiter(
+ (long)
+ IoTDBDescriptor.getInstance()
+ .getConfig()
+ .getPartitionTableRecoverMaxReadBytesPerSecond()
+ * 1024
+ * 1024);
+
+ public static final String SCAN_FILE_SUFFIX_NAME = ".tsfile";
+ public static final Set<String> IGNORE_DATABASE =
+ new HashSet<String>() {
+ {
+ add("root.__audit");
+ add("root.__system");
+ }
+ };
+
+ public DataPartitionTableGenerator(
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String dataDirectory,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = new String[] {dataDirectory};
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String[] dataDirectories,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = dataDirectories;
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public enum TaskStatus {
+ NOT_STARTED,
+ IN_PROGRESS,
+ COMPLETED,
+ FAILED
+ }
+
+ /** Start generating DataPartitionTable asynchronously. */
+ public CompletableFuture<Void> startGeneration() {
+ if (status != TaskStatus.NOT_STARTED) {
+ throw new IllegalStateException("Task is already started or completed");
+ }
+
+ status = TaskStatus.IN_PROGRESS;
+ return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);
Review Comment:
`startGeneration()` uses `CompletableFuture.runAsync(...)` without an executor, so it runs on the common ForkJoinPool rather than the `executor` passed to the generator. This can cause unexpected thread usage/priority and makes throttling harder; pass an explicit executor and ensure failures set `status=FAILED` + `errorMessage`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3158,292 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
Review Comment:
New RPC methods are introduced here, but no tests are added to validate their behavior despite existing test infrastructure for `DataNodeInternalRPCServiceImpl`. Adding tests for directory scanning and DataPartitionTable ser/de (including empty/missing paths and timeout handling) would reduce regression risk.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -1219,6 +1219,10 @@ public class IoTDBConfig {
private long maxObjectSizeInByte = 4 * 1024 * 1024 * 1024L;
+ /* Need use these parameters when repair data partition table */
+ private int partitionTableRecoverWorkerNum = 10;
+ private int partitionTableRecoverMaxReadBytesPerSecond = 1000;
Review Comment:
The default `partitionTableRecoverMaxReadBytesPerSecond` here is `1000`, but the config template (`iotdb-system.properties.template`) documents a default of `10` (MB/s). Keeping code defaults and the shipped template defaults inconsistent is confusing and can lead to unexpected behavior; align the defaults (or document why they intentionally differ).
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+ }
+
+ if (finalDataPartitionMap.isEmpty()) {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (dnSeriesPartitionSlot == null || dnSeriesPartitionTable == null) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ dnSeriesPartitionSlot, k -> dnSeriesPartitionTable);
+ });
+ });
+ } else {
+ finalDataPartitionMap.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTable) -> {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (!tSeriesPartitionSlot.equals(dnSeriesPartitionSlot)) {
+ return;
+ }
+
+ if (seriesPartitionTable == null
+ || seriesPartitionTable.getSeriesPartitionMap() == null
+ || seriesPartitionTable.getSeriesPartitionMap().isEmpty()) {
+ finalDataPartitionMap.put(
+ tSeriesPartitionSlot, dnSeriesPartitionTable);
+ }
+
+ // dnDataPartitionTable merged to seriesPartitionTable
+ dnSeriesPartitionTable
+ .getSeriesPartitionMap()
+ .forEach(
+ (k, v) ->
+ v.forEach(
+ tConsensusGroupId -> {
+ if (seriesPartitionTable == null) {
+ return;
+ }
+ seriesPartitionTable.putDataPartition(
+ k, tConsensusGroupId);
+ }));
+ });
+ });
+ });
+ }
+
+ finalDataPartitionTable = new DataPartitionTable(finalDataPartitionMap);
+
+ LOG.info("DataPartitionTable merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTable == null) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
+ try {
+ CreateDataPartitionPlan createPlan = new CreateDataPartitionPlan();
+ Map<String, DataPartitionTable> assignedDataPartition = new HashMap<>();
+ assignedDataPartition.put(
+ lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);
+ createPlan.setAssignedDataPartition(assignedDataPartition);
+ TSStatus tsStatus = env.getConfigManager().getConsensusManager().write(createPlan);
+
+ if (tsStatus.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.info("DataPartitionTable successfully written to raft log");
+ break;
+ } else {
+ LOG.error("Failed to write DataPartitionTable to raft log");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("Failed to write DataPartitionTable to raft log"));
+ }
+ } catch (Exception e) {
+ LOG.error("Error writing DataPartitionTable to raft log", e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ }
+ failedCnt++;
+ }
+
+ return getFlow();
+ }
+
+ private Flow getFlow() {
+ if (!failedDataNodes.isEmpty()) {
+ allDataNodes.removeAll(failedDataNodes);
+ skipDataNodes = new HashSet<>(allDataNodes);
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ } else {
+ skipDataNodes.clear();
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ public void serialize(final DataOutputStream stream) throws IOException {
+ super.serialize(stream);
+
+ // Serialize earliestTimeslots
+ stream.writeInt(earliestTimeslots.size());
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ stream.writeUTF(entry.getKey());
+ stream.writeLong(entry.getValue());
+ }
+
+ // Serialize dataPartitionTables count
+ stream.writeInt(dataPartitionTables.size());
+ for (Map.Entry<Integer, DataPartitionTable> entry : dataPartitionTables.entrySet()) {
+ stream.writeInt(entry.getKey());
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ entry.getValue().serialize(oos, protocol);
+
+ // Write the size and data for byte array after serialize
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize dataPartitionTables", e);
+ }
+ }
+
+ stream.writeInt(lostDataPartitionsOfDatabases.size());
+ for (String database : lostDataPartitionsOfDatabases) {
+ stream.writeUTF(database);
Review Comment:
`lostDataPartitionsOfDatabases` is written with `writeUTF`, but the corresponding `deserialize()` path uses `ReadWriteIOUtils.readString(...)` (different encoding/length). This will desynchronize the remaining fields and break recovery; use the same codec for both serialization and deserialization.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+ }
+
+ if (finalDataPartitionMap.isEmpty()) {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (dnSeriesPartitionSlot == null || dnSeriesPartitionTable == null) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ dnSeriesPartitionSlot, k -> dnSeriesPartitionTable);
+ });
+ });
+ } else {
+ finalDataPartitionMap.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTable) -> {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (!tSeriesPartitionSlot.equals(dnSeriesPartitionSlot)) {
+ return;
+ }
+
+ if (seriesPartitionTable == null
+ || seriesPartitionTable.getSeriesPartitionMap() == null
+ || seriesPartitionTable.getSeriesPartitionMap().isEmpty()) {
+ finalDataPartitionMap.put(
+ tSeriesPartitionSlot, dnSeriesPartitionTable);
+ }
+
+ // dnDataPartitionTable merged to seriesPartitionTable
+ dnSeriesPartitionTable
+ .getSeriesPartitionMap()
+ .forEach(
+ (k, v) ->
+ v.forEach(
+ tConsensusGroupId -> {
+ if (seriesPartitionTable == null) {
+ return;
+ }
+ seriesPartitionTable.putDataPartition(
+ k, tConsensusGroupId);
+ }));
+ });
+ });
+ });
+ }
+
+ finalDataPartitionTable = new DataPartitionTable(finalDataPartitionMap);
+
+ LOG.info("DataPartitionTable merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTable == null) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
+ try {
+ CreateDataPartitionPlan createPlan = new CreateDataPartitionPlan();
+ Map<String, DataPartitionTable> assignedDataPartition = new HashMap<>();
+ assignedDataPartition.put(
+ lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);
+ createPlan.setAssignedDataPartition(assignedDataPartition);
+ TSStatus tsStatus = env.getConfigManager().getConsensusManager().write(createPlan);
+
+ if (tsStatus.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.info("DataPartitionTable successfully written to raft log");
+ break;
+ } else {
+ LOG.error("Failed to write DataPartitionTable to raft log");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("Failed to write DataPartitionTable to raft log"));
+ }
+ } catch (Exception e) {
+ LOG.error("Error writing DataPartitionTable to raft log", e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ }
+ failedCnt++;
+ }
+
+ return getFlow();
+ }
+
+ private Flow getFlow() {
+ if (!failedDataNodes.isEmpty()) {
+ allDataNodes.removeAll(failedDataNodes);
+ skipDataNodes = new HashSet<>(allDataNodes);
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ } else {
+ skipDataNodes.clear();
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ public void serialize(final DataOutputStream stream) throws IOException {
+ super.serialize(stream);
+
+ // Serialize earliestTimeslots
+ stream.writeInt(earliestTimeslots.size());
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ stream.writeUTF(entry.getKey());
+ stream.writeLong(entry.getValue());
+ }
+
+ // Serialize dataPartitionTables count
+ stream.writeInt(dataPartitionTables.size());
+ for (Map.Entry<Integer, DataPartitionTable> entry : dataPartitionTables.entrySet()) {
+ stream.writeInt(entry.getKey());
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ entry.getValue().serialize(oos, protocol);
+
+ // Write the size and data for byte array after serialize
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize dataPartitionTables", e);
+ }
+ }
+
+ stream.writeInt(lostDataPartitionsOfDatabases.size());
+ for (String database : lostDataPartitionsOfDatabases) {
+ stream.writeUTF(database);
+ }
+
+ if (finalDataPartitionTable != null) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ finalDataPartitionTable.serialize(oos, protocol);
+
+ // Write the size and data for byte array after serialize
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize finalDataPartitionTable", e);
+ }
+ } else {
+ stream.writeInt(0);
+ }
+
+ stream.writeInt(skipDataNodes.size());
+ for (TDataNodeConfiguration skipDataNode : skipDataNodes) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
+ TTransport transport = new TIOStreamTransport(baos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ skipDataNode.write(protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (TException e) {
+ LOG.error("Failed to serialize skipDataNode", e);
+ throw new IOException("Failed to serialize skipDataNode", e);
+ }
+ }
+
+ stream.writeInt(failedDataNodes.size());
+ for (TDataNodeConfiguration failedDataNode : failedDataNodes) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
+ TTransport transport = new TIOStreamTransport(baos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ failedDataNode.write(protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (TException e) {
+ LOG.error("Failed to serialize failedDataNode", e);
+ throw new IOException("Failed to serialize failedDataNode", e);
+ }
+ }
+ }
+
+ @Override
+ public void deserialize(final ByteBuffer byteBuffer) {
+ super.deserialize(byteBuffer);
+
+ // Deserialize earliestTimeslots
+ int earliestTimeslotsSize = byteBuffer.getInt();
+ earliestTimeslots = new ConcurrentHashMap<>();
+ for (int i = 0; i < earliestTimeslotsSize; i++) {
+ String database = String.valueOf(byteBuffer.getChar());
Review Comment:
Procedure serialization is broken: `serialize()` writes database names with `writeUTF`, but `deserialize()` reads them via `byteBuffer.getChar()` (one character). This will corrupt the buffer and make procedure recovery fail. Use a matching string codec on both sides (e.g., `ReadWriteIOUtils.writeString` + `readString`, consistently).
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/ConfigNodeProcedureEnv.java:
##########
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.confignode.manager.ConfigManager;
+
+/**
+ * Environment object for ConfigNode procedures. Provides access to ConfigManager and other
+ * necessary components.
+ */
+public class ConfigNodeProcedureEnv {
+
+ private final ConfigManager configManager;
+
+ public ConfigNodeProcedureEnv(ConfigManager configManager) {
+ this.configManager = configManager;
+ }
+
+ public ConfigManager getConfigManager() {
+ return configManager;
+ }
Review Comment:
This class duplicates the existing `org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv`, while the new procedure currently imports the existing env type (not this one). Having two env classes with the same name in different packages is confusing and this one appears unused; consider removing it or using it consistently.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupType;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.partition.executor.SeriesPartitionExecutor;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.commons.utils.rateLimiter.LeakyBucketRateLimiter;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Generator for DataPartitionTable by scanning tsfile resources. This class scans the data
+ * directory structure and builds a complete DataPartitionTable based on existing tsfiles.
+ */
+public class DataPartitionTableGenerator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DataPartitionTableGenerator.class);
+
+ // Task status
+ private volatile TaskStatus status = TaskStatus.NOT_STARTED;
+ private volatile String errorMessage;
+ private volatile DataPartitionTable dataPartitionTable;
+
+ // Progress tracking
+ private final AtomicInteger processedFiles = new AtomicInteger(0);
+ private final AtomicInteger failedFiles = new AtomicInteger(0);
+ private final AtomicLong totalFiles = new AtomicLong(0);
+
+ // Configuration
+ private String[] dataDirectories;
+ private final ExecutorService executor;
+ private final Set<String> databases;
+ private final int seriesSlotNum;
+ private final String seriesPartitionExecutorClass;
+
+ private static final int EXECUTOR_MAX_TIMEOUT = 60;
+
+ private static final LeakyBucketRateLimiter limiter =
+ new LeakyBucketRateLimiter(
+ (long)
+ IoTDBDescriptor.getInstance()
+ .getConfig()
+ .getPartitionTableRecoverMaxReadBytesPerSecond()
+ * 1024
+ * 1024);
+
+ public static final String SCAN_FILE_SUFFIX_NAME = ".tsfile";
+ public static final Set<String> IGNORE_DATABASE =
+ new HashSet<String>() {
+ {
+ add("root.__audit");
+ add("root.__system");
+ }
+ };
+
+ public DataPartitionTableGenerator(
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String dataDirectory,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = new String[] {dataDirectory};
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String[] dataDirectories,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = dataDirectories;
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public enum TaskStatus {
+ NOT_STARTED,
+ IN_PROGRESS,
+ COMPLETED,
+ FAILED
+ }
+
+ /** Start generating DataPartitionTable asynchronously. */
+ public CompletableFuture<Void> startGeneration() {
+ if (status != TaskStatus.NOT_STARTED) {
+ throw new IllegalStateException("Task is already started or completed");
+ }
+
+ status = TaskStatus.IN_PROGRESS;
+ return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);
+ }
+
+ private void generateDataPartitionTableByMemory() {
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap = new ConcurrentHashMap<>();
+ List<CompletableFuture<Void>> futures = new ArrayList<>();
+
+ SeriesPartitionExecutor seriesPartitionExecutor =
+ SeriesPartitionExecutor.getSeriesPartitionExecutor(
+ seriesPartitionExecutorClass, seriesSlotNum);
+
+ for (DataRegion dataRegion : StorageEngine.getInstance().getAllDataRegions()) {
+ CompletableFuture<Void> regionFuture =
+ CompletableFuture.runAsync(
+ () -> {
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ String databaseName = dataRegion.getDatabaseName();
+ if (!databases.contains(databaseName) || IGNORE_DATABASE.contains(databaseName)) {
+ return;
+ }
+
+ tsFileManager.readLock();
+ List<TsFileResource> seqTsFileList = tsFileManager.getTsFileList(true);
+ List<TsFileResource> unseqTsFileList = tsFileManager.getTsFileList(false);
+ tsFileManager.readUnlock();
+
+ constructDataPartitionMap(seqTsFileList, seriesPartitionExecutor, dataPartitionMap);
+ constructDataPartitionMap(
+ unseqTsFileList, seriesPartitionExecutor, dataPartitionMap);
+ },
+ executor);
+ futures.add(regionFuture);
+ }
+
+ // Wait for all tasks to complete
+ CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).join();
+
+ if (dataPartitionMap.isEmpty()) {
+ LOG.error("Failed to generate DataPartitionTable, dataPartitionMap is empty");
+ status = TaskStatus.FAILED;
+ return;
+ }
+
+ dataPartitionTable = new DataPartitionTable(dataPartitionMap);
+ status = TaskStatus.COMPLETED;
+ }
+
+ private static void constructDataPartitionMap(
+ List<TsFileResource> seqTsFileList,
+ SeriesPartitionExecutor seriesPartitionExecutor,
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> dataPartitionMap) {
+ for (TsFileResource tsFileResource : seqTsFileList) {
+ Set<IDeviceID> devices = tsFileResource.getDevices(limiter);
+ long timeSlotId = tsFileResource.getTsFileID().timePartitionId;
+ int regionId = tsFileResource.getTsFileID().regionId;
+
+ TConsensusGroupId consensusGroupId = new TConsensusGroupId();
+ consensusGroupId.setId(regionId);
+ consensusGroupId.setType(TConsensusGroupType.DataRegion);
+
+ for (IDeviceID deviceId : devices) {
+ TSeriesPartitionSlot seriesSlotId =
+ seriesPartitionExecutor.getSeriesPartitionSlot(deviceId);
+ TTimePartitionSlot timePartitionSlot =
+ new TTimePartitionSlot(TimePartitionUtils.getTimeByPartitionId(timeSlotId));
+ dataPartitionMap
+ .computeIfAbsent(
+ seriesSlotId, empty -> newSeriesPartitionTable(consensusGroupId, timeSlotId))
Review Comment:
This `computeIfAbsent(... newSeriesPartitionTable(...))` path constructs a `SeriesPartitionTable` that already contains the first mapping, and then immediately calls `putDataPartition(...)` for the same slot/group again. This redundant double-insert complicates correctness (duplicates) and adds overhead; insert the mapping only once.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -203,6 +221,34 @@ public void active() {
}
loadSecretKey();
loadHardwareCode();
+
+ dataPartitionTableCheckFuture =
+ dataPartitionTableCheckExecutor.submit(
+ () -> {
+ LOGGER.info(
+ "Prepare to start dataPartitionTableIntegrityCheck after all datanodes are started up");
+ // Thread.sleep(CONF.getPartitionTableRecoverWaitAllDnUpTimeout());
+
+ while (latch.getCount() > 0) {
Review Comment:
The integrity-check startup task currently ignores the configured `partition_table_recover_wait_all_dn_up_timeout` (the `Thread.sleep(...)` is commented out) and instead loops until it finds a running DataNode. This can wait forever; use the configured timeout (or a bounded wait) and fail/skip cleanly when the timeout elapses.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/service/ConfigNode.java:
##########
@@ -147,6 +160,11 @@ protected void start() throws IoTDBException {
}
active();
LOGGER.info("IoTDB started");
+ try {
+ dataPartitionTableCheckFuture.get();
+ } catch (ExecutionException | InterruptedException e) {
+ LOGGER.error("Data partition table check task execute failed", e);
Review Comment:
`start()` calls `dataPartitionTableCheckFuture.get()` unconditionally, but `dataPartitionTableCheckFuture` is only assigned in the restart branch of `active()`. This can throw NPE on first startup and also blocks the main startup thread waiting for a best-effort background task. Guard against null and avoid blocking startup here.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -4367,4 +4371,20 @@ public long getMaxObjectSizeInByte() {
public void setMaxObjectSizeInByte(long maxObjectSizeInByte) {
this.maxObjectSizeInByte = maxObjectSizeInByte;
}
+
+ public int getPartitionTableRecoverWorkerNum() {
+ return partitionTableRecoverWorkerNum;
+ }
+
+ public void setPartitionTableRecoverWorkerNum(int partitionTableRecoverWorkerNum) {
+ this.partitionTableRecoverWorkerNum = partitionTableRecoverWorkerNum;
+ }
+
+ public int getPartitionTableRecoverMaxReadBytesPerSecond() {
+ return partitionTableRecoverMaxReadBytesPerSecond;
+ }
+
+ public void setPartitionTableRecoverMaxReadBytesPerSecond(int partitionTableRecoverWorkerNum) {
+ this.partitionTableRecoverWorkerNum = partitionTableRecoverWorkerNum;
Review Comment:
`setPartitionTableRecoverMaxReadBytesPerSecond` assigns to `partitionTableRecoverWorkerNum` instead of `partitionTableRecoverMaxReadBytesPerSecond`, so the configured read rate is never applied and the worker count may be accidentally overwritten. Update the setter to write to the correct field and fix the parameter name accordingly.
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/manager/node/NodeManager.java:
##########
@@ -1346,4 +1349,56 @@ private TTLManager getTTLManager() {
private ExternalServiceManager getServiceManager() {
return configManager.getExternalServiceManager();
}
+
+ /**
+ * Check if all DataNodes are registered and running, then trigger integrity check. This method
+ * should be called after each DataNode registration.
+ */
+ private void checkAndTriggerIntegrityCheck() {
+ // Only trigger integrity check if this ConfigNode is the leader
+ if (!configManager.getConsensusManager().isLeader()) {
+ return;
+ }
+
+ // Get all registered DataNodes
+ List<TDataNodeConfiguration> registeredDataNodes = getRegisteredDataNodes();
+
+ // Check if all registered DataNodes are running
+ boolean allDataNodesRunning =
+ registeredDataNodes.stream()
+ .allMatch(
+ dataNode -> {
+ Integer dataNodeId = dataNode.getLocation().getDataNodeId();
+ NodeStatus status = getLoadManager().getLoadCache().getNodeStatus(dataNodeId);
+ return status == NodeStatus.Running;
+ });
+
+ if (allDataNodesRunning && !registeredDataNodes.isEmpty()) {
+ LOGGER.info(
+ "All {} DataNodes are registered and running, triggering data partition table integrity check",
+ registeredDataNodes.size());
+
+ // Trigger integrity check asynchronously
+ try {
+ configManager.getProcedureManager().dataPartitionTableIntegrityCheck();
+ LOGGER.info("Data partition table integrity check procedure submitted successfully");
Review Comment:
The comment says "Trigger integrity check asynchronously", but `dataPartitionTableIntegrityCheck()` waits for the procedure to finish, so this call is actually blocking. Either make `dataPartitionTableIntegrityCheck()` non-blocking (submit-only) or update this call site to run it truly asynchronously (e.g., via a background executor) to avoid stalling registration/startup flows.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/rateLimiter/LeakyBucketRateLimiter.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.commons.utils.rateLimiter;
+
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.locks.LockSupport;
+
+/**
+ * A global leaky-bucket rate limiter for bytes throughput. Features: - Strict throughput limiting
+ * (no burst) - Smooth bandwidth shaping - Thread-safe - Fair for multi-thread - Low contention
+ */
+public class LeakyBucketRateLimiter {
+ /** bytes per second */
+ private volatile long bytesPerSecond;
+
+ /** start time */
+ private final long startTimeNs;
+
+ /** total consumed bytes */
+ private final AtomicLong totalBytes = new AtomicLong(0);
+
+ public LeakyBucketRateLimiter(long bytesPerSecond) {
+ if (bytesPerSecond <= 0) {
+ throw new IllegalArgumentException("bytesPerSecond must be > 0");
+ }
+ this.bytesPerSecond = bytesPerSecond;
+ this.startTimeNs = System.nanoTime();
+ }
+
+ /**
+ * Acquire permission for reading bytes.
+ *
+ * <p>This method will block if reading too fast.
+ */
+ public void acquire(long bytes) {
+ if (bytes <= 0) {
+ return;
+ }
+
+ long currentTotal = totalBytes.addAndGet(bytes);
+
+ long expectedTimeNs = expectedTimeNs(currentTotal);
+ long now = System.nanoTime();
+
+ long sleepNs = expectedTimeNs - now;
+
+ if (sleepNs > 0) {
+ LockSupport.parkNanos(sleepNs);
+ }
+ }
+
+ /**
+ * Try acquire without blocking.
+ *
+ * @return true if allowed immediately
+ */
+ public boolean tryAcquire(long bytes) {
+ if (bytes <= 0) {
+ return true;
+ }
+
+ long currentTotal = totalBytes.addAndGet(bytes);
+
+ long expectedTimeNs = expectedTimeNs(currentTotal);
+ long now = System.nanoTime();
+
+ if (expectedTimeNs <= now) {
+ return true;
+ }
+
+ // rollback
+ totalBytes.addAndGet(-bytes);
+ return false;
+ }
+
+ /** Update rate dynamically. */
+ public void setRate(long newBytesPerSecond) {
+ if (newBytesPerSecond <= 0) {
+ throw new IllegalArgumentException("bytesPerSecond must be > 0");
+ }
+ this.bytesPerSecond = newBytesPerSecond;
+ }
+
+ /** Current rate. */
+ public long getRate() {
+ return bytesPerSecond;
+ }
+
+ /** Total bytes processed. */
+ public long getTotalBytes() {
+ return totalBytes.get();
+ }
+
+ /** Expected time based on bytes processed. */
+ private long expectedTimeNs(long totalBytes) {
+ return startTimeNs + (totalBytes * 1_000_000_000L) / bytesPerSecond;
Review Comment:
`expectedTimeNs` computes `(totalBytes * 1_000_000_000L) / bytesPerSecond`, which can overflow `long` for large `totalBytes` and produce negative/incorrect sleep times. Consider using `Math.multiplyExact` with a fallback (or BigInteger) and clamping, and/or track time in smaller units to avoid overflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944162273
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/partition/DataPartitionTableGenerator.java:
##########
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupType;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.partition.executor.SeriesPartitionExecutor;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.commons.utils.rateLimiter.LeakyBucketRateLimiter;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Generator for DataPartitionTable by scanning tsfile resources. This class scans the data
+ * directory structure and builds a complete DataPartitionTable based on existing tsfiles.
+ */
+public class DataPartitionTableGenerator {
+
+ private static final Logger LOG = LoggerFactory.getLogger(DataPartitionTableGenerator.class);
+
+ // Task status
+ private volatile TaskStatus status = TaskStatus.NOT_STARTED;
+ private volatile String errorMessage;
+ private volatile DataPartitionTable dataPartitionTable;
+
+ // Progress tracking
+ private final AtomicInteger processedFiles = new AtomicInteger(0);
+ private final AtomicInteger failedFiles = new AtomicInteger(0);
+ private final AtomicLong totalFiles = new AtomicLong(0);
+
+ // Configuration
+ private String[] dataDirectories;
+ private final ExecutorService executor;
+ private final Set<String> databases;
+ private final int seriesSlotNum;
+ private final String seriesPartitionExecutorClass;
+
+ private static final int EXECUTOR_MAX_TIMEOUT = 60;
+
+ private static final LeakyBucketRateLimiter limiter =
+ new LeakyBucketRateLimiter(
+ (long)
+ IoTDBDescriptor.getInstance()
+ .getConfig()
+ .getPartitionTableRecoverMaxReadBytesPerSecond()
+ * 1024
+ * 1024);
+
+ public static final String SCAN_FILE_SUFFIX_NAME = ".tsfile";
+ public static final Set<String> IGNORE_DATABASE =
+ new HashSet<String>() {
+ {
+ add("root.__audit");
+ add("root.__system");
+ }
+ };
+
+ public DataPartitionTableGenerator(
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String dataDirectory,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = new String[] {dataDirectory};
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public DataPartitionTableGenerator(
+ String[] dataDirectories,
+ ExecutorService executor,
+ Set<String> databases,
+ int seriesSlotNum,
+ String seriesPartitionExecutorClass) {
+ this.dataDirectories = dataDirectories;
+ this.executor = executor;
+ this.databases = databases;
+ this.seriesSlotNum = seriesSlotNum;
+ this.seriesPartitionExecutorClass = seriesPartitionExecutorClass;
+ }
+
+ public enum TaskStatus {
+ NOT_STARTED,
+ IN_PROGRESS,
+ COMPLETED,
+ FAILED
+ }
+
+ /** Start generating DataPartitionTable asynchronously. */
+ public CompletableFuture<Void> startGeneration() {
+ if (status != TaskStatus.NOT_STARTED) {
+ throw new IllegalStateException("Task is already started or completed");
+ }
+
+ status = TaskStatus.IN_PROGRESS;
+ return CompletableFuture.runAsync(this::generateDataPartitionTableByMemory);
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944158863
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -414,6 +427,34 @@ public class DataNodeInternalRPCServiceImpl implements IDataNodeRPCService.Iface
private static final String SYSTEM = "system";
+ private final ExecutorService findEarliestTimeSlotExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName()),
+ ThreadName.FIND_EARLIEST_TIME_SLOT_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private final ExecutorService partitionTableRecoverExecutor =
+ new WrappedThreadPoolExecutor(
+ 0,
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum(),
+ 0L,
+ TimeUnit.SECONDS,
+ new ArrayBlockingQueue<>(
+ IoTDBDescriptor.getInstance().getConfig().getPartitionTableRecoverWorkerNum()),
+ new IoTThreadFactory(ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName()),
+ ThreadName.DATA_PARTITION_RECOVER_PARALLEL_POOL.getName(),
+ new ThreadPoolExecutor.CallerRunsPolicy());
+
+ private Map<String, Long> databaseEarliestRegionMap = new ConcurrentHashMap<>();
+
Review Comment:
Igonre.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944672219
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java:
##########
@@ -1219,6 +1219,10 @@ public class IoTDBConfig {
private long maxObjectSizeInByte = 4 * 1024 * 1024 * 1024L;
+ /* Need use these parameters when repair data partition table */
+ private int partitionTableRecoverWorkerNum = 10;
+ private int partitionTableRecoverMaxReadBytesPerSecond = 1000;
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2954307486
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,880 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.DatabaseScopedDataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 10000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, List<DatabaseScopedDataPartitionTable>> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private Map<String, DataPartitionTable> finalDataPartitionTables;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = Collections.newSetFromMap(new ConcurrentHashMap<>());
+ private static Set<TDataNodeConfiguration> failedDataNodes =
+ Collections.newSetFromMap(new ConcurrentHashMap<>());
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ return requestPartitionTables();
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ return requestPartitionTablesHeartBeat();
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables = new HashMap<>();
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ // Cleanup resources
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ earliestTimeslots.clear();
+ break;
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases.clear();
+ break;
+ case REQUEST_PARTITION_TABLES:
+ case REQUEST_PARTITION_TABLES_HEART_BEAT:
+ dataPartitionTables.clear();
+ break;
+ case MERGE_PARTITION_TABLES:
+ finalDataPartitionTables.clear();
+ break;
+ default:
+ allDataNodes.clear();
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ finalDataPartitionTables.clear();
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, waiting for them to go up");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.warn(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+
+ if (tTimePartitionSlotListMap.isEmpty()) {
+ continue;
+ }
+
+ TTimePartitionSlot localEarliestSlot = tTimePartitionSlotListMap.keySet()
+ .stream()
+ .min(Comparator.comparingLong(TTimePartitionSlot::getStartTime))
+ .orElse(null);
+
+ if (localEarliestSlot.getStartTime() > TimePartitionUtils.getTimeByPartitionId(earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ }
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Flow requestPartitionTablesHeartBeat() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+
+ switch (state) {
+ case SUCCESS:
+ List<ByteBuffer> byteBufferList = resp.getDatabaseScopedDataPartitionTables();
+ List<DatabaseScopedDataPartitionTable> databaseScopedDataPartitionTableList = deserializeDatabaseScopedTableList(byteBufferList);
+ dataPartitionTables.put(dataNodeId, databaseScopedDataPartitionTableList);
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ try {
+ Thread.sleep(HEART_BEAT_REQUEST_RATE);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ LOG.error("Error checking DataPartitionTable status due to thread interruption.");
+ }
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES_HEART_BEAT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+
+ dataPartitionTables.forEach((k, v) -> v.forEach(databaseScopedDataPartitionTable -> {
+ if (!databaseScopedDataPartitionTable.getDatabase().equals(database)) {
+ return;
+ }
+ finalDataPartitionTables.put(database, new DataPartitionTable(finalDataPartitionMap).merge(databaseScopedDataPartitionTable.getDataPartitionTable()));
+ }));
+ }
+
+ LOG.info("DataPartitionTables merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944045205
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
Review Comment:
You can see two functions have been added to the DataPartitionTable class.
<img width="1556" height="984" alt="image" src="https://github.com/user-attachments/assets/f7c13b13-19e8-472c-97ae-9036feffd1a4" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944101073
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+ }
+
+ if (finalDataPartitionMap.isEmpty()) {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (dnSeriesPartitionSlot == null || dnSeriesPartitionTable == null) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ dnSeriesPartitionSlot, k -> dnSeriesPartitionTable);
+ });
+ });
+ } else {
+ finalDataPartitionMap.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTable) -> {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (!tSeriesPartitionSlot.equals(dnSeriesPartitionSlot)) {
+ return;
+ }
+
+ if (seriesPartitionTable == null
+ || seriesPartitionTable.getSeriesPartitionMap() == null
+ || seriesPartitionTable.getSeriesPartitionMap().isEmpty()) {
+ finalDataPartitionMap.put(
+ tSeriesPartitionSlot, dnSeriesPartitionTable);
+ }
+
+ // dnDataPartitionTable merged to seriesPartitionTable
+ dnSeriesPartitionTable
+ .getSeriesPartitionMap()
+ .forEach(
+ (k, v) ->
+ v.forEach(
+ tConsensusGroupId -> {
+ if (seriesPartitionTable == null) {
+ return;
+ }
+ seriesPartitionTable.putDataPartition(
+ k, tConsensusGroupId);
+ }));
+ });
+ });
+ });
+ }
+
+ finalDataPartitionTable = new DataPartitionTable(finalDataPartitionMap);
+
+ LOG.info("DataPartitionTable merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTable == null) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
+ try {
+ CreateDataPartitionPlan createPlan = new CreateDataPartitionPlan();
+ Map<String, DataPartitionTable> assignedDataPartition = new HashMap<>();
+ assignedDataPartition.put(
+ lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);
+ createPlan.setAssignedDataPartition(assignedDataPartition);
+ TSStatus tsStatus = env.getConfigManager().getConsensusManager().write(createPlan);
+
+ if (tsStatus.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.info("DataPartitionTable successfully written to raft log");
+ break;
+ } else {
+ LOG.error("Failed to write DataPartitionTable to raft log");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("Failed to write DataPartitionTable to raft log"));
+ }
+ } catch (Exception e) {
+ LOG.error("Error writing DataPartitionTable to raft log", e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ }
+ failedCnt++;
+ }
+
+ return getFlow();
+ }
+
+ private Flow getFlow() {
+ if (!failedDataNodes.isEmpty()) {
+ allDataNodes.removeAll(failedDataNodes);
+ skipDataNodes = new HashSet<>(allDataNodes);
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ } else {
+ skipDataNodes.clear();
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ public void serialize(final DataOutputStream stream) throws IOException {
+ super.serialize(stream);
+
+ // Serialize earliestTimeslots
+ stream.writeInt(earliestTimeslots.size());
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ stream.writeUTF(entry.getKey());
+ stream.writeLong(entry.getValue());
+ }
+
+ // Serialize dataPartitionTables count
+ stream.writeInt(dataPartitionTables.size());
+ for (Map.Entry<Integer, DataPartitionTable> entry : dataPartitionTables.entrySet()) {
+ stream.writeInt(entry.getKey());
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ entry.getValue().serialize(oos, protocol);
+
+ // Write the size and data for byte array after serialize
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize dataPartitionTables", e);
+ }
+ }
+
+ stream.writeInt(lostDataPartitionsOfDatabases.size());
+ for (String database : lostDataPartitionsOfDatabases) {
+ stream.writeUTF(database);
+ }
+
+ if (finalDataPartitionTable != null) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ finalDataPartitionTable.serialize(oos, protocol);
+
+ // Write the size and data for byte array after serialize
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize finalDataPartitionTable", e);
+ }
+ } else {
+ stream.writeInt(0);
+ }
+
+ stream.writeInt(skipDataNodes.size());
+ for (TDataNodeConfiguration skipDataNode : skipDataNodes) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
+ TTransport transport = new TIOStreamTransport(baos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ skipDataNode.write(protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (TException e) {
+ LOG.error("Failed to serialize skipDataNode", e);
+ throw new IOException("Failed to serialize skipDataNode", e);
+ }
+ }
+
+ stream.writeInt(failedDataNodes.size());
+ for (TDataNodeConfiguration failedDataNode : failedDataNodes) {
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
+ TTransport transport = new TIOStreamTransport(baos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ failedDataNode.write(protocol);
+
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (TException e) {
+ LOG.error("Failed to serialize failedDataNode", e);
+ throw new IOException("Failed to serialize failedDataNode", e);
+ }
+ }
+ }
+
+ @Override
+ public void deserialize(final ByteBuffer byteBuffer) {
+ super.deserialize(byteBuffer);
+
+ // Deserialize earliestTimeslots
+ int earliestTimeslotsSize = byteBuffer.getInt();
+ earliestTimeslots = new ConcurrentHashMap<>();
+ for (int i = 0; i < earliestTimeslotsSize; i++) {
+ String database = String.valueOf(byteBuffer.getChar());
Review Comment:
Changed yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944197556
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/utils/TimePartitionUtils.java:
##########
@@ -122,6 +130,23 @@ public static long getTimePartitionIdWithoutOverflow(long time) {
return partitionId.longValue();
}
+ public static long getTimeWithoutOverflow(long partitionId) {
+ BigInteger bigTime = bigTimePartitionInterval.multiply(BigInteger.valueOf(partitionId));
+ if (bigTime.compareTo(BigInteger.ZERO) > 0
+ || bigTime.remainder(bigTimePartitionInterval).equals(BigInteger.ZERO)) {
+ return bigTime.add(bigTimePartitionOrigin).longValue();
+ }
+ return BigInteger.valueOf(partitionId)
+ .add(BigInteger.ONE)
+ .multiply(bigTimePartitionInterval)
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944657056
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2944114340
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/partition/DataPartitionTableIntegrityCheckProcedure.java:
##########
@@ -0,0 +1,878 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.confignode.procedure.impl.partition;
+
+import org.apache.iotdb.common.rpc.thrift.TConsensusGroupId;
+import org.apache.iotdb.common.rpc.thrift.TDataNodeConfiguration;
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSeriesPartitionSlot;
+import org.apache.iotdb.common.rpc.thrift.TTimePartitionSlot;
+import org.apache.iotdb.commons.concurrent.threadpool.ScheduledExecutorUtil;
+import org.apache.iotdb.commons.enums.DataPartitionTableGeneratorState;
+import org.apache.iotdb.commons.partition.DataPartitionTable;
+import org.apache.iotdb.commons.partition.SeriesPartitionTable;
+import org.apache.iotdb.commons.utils.TimePartitionUtils;
+import org.apache.iotdb.confignode.client.sync.CnToDnSyncRequestType;
+import org.apache.iotdb.confignode.client.sync.SyncDataNodeClientPool;
+import org.apache.iotdb.confignode.consensus.request.read.partition.GetDataPartitionPlan;
+import org.apache.iotdb.confignode.consensus.request.write.partition.CreateDataPartitionPlan;
+import org.apache.iotdb.confignode.manager.node.NodeManager;
+import org.apache.iotdb.confignode.procedure.env.ConfigNodeProcedureEnv;
+import org.apache.iotdb.confignode.procedure.exception.ProcedureException;
+import org.apache.iotdb.confignode.procedure.impl.StateMachineProcedure;
+import org.apache.iotdb.confignode.procedure.state.DataPartitionTableIntegrityCheckProcedureState;
+import org.apache.iotdb.confignode.rpc.thrift.TTimeSlotList;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableHeartbeatResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableReq;
+import org.apache.iotdb.mpp.rpc.thrift.TGenerateDataPartitionTableResp;
+import org.apache.iotdb.mpp.rpc.thrift.TGetEarliestTimeslotsResp;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.transport.TIOStreamTransport;
+import org.apache.thrift.transport.TTransport;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectOutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * Procedure for checking and restoring data partition table integrity. This procedure scans all
+ * DataNodes to detect missing data partitions and restores the DataPartitionTable on the ConfigNode
+ * Leader.
+ */
+public class DataPartitionTableIntegrityCheckProcedure
+ extends StateMachineProcedure<
+ ConfigNodeProcedureEnv, DataPartitionTableIntegrityCheckProcedureState> {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(DataPartitionTableIntegrityCheckProcedure.class);
+
+ private static final int MAX_RETRY_COUNT = 3;
+ private static final long HEART_BEAT_REQUEST_RATE = 60000;
+
+ NodeManager dataNodeManager;
+ private List<TDataNodeConfiguration> allDataNodes = new ArrayList<>();
+
+ // ============Need serialize BEGIN=============/
+ /** Collected earliest timeslots from DataNodes: database -> earliest timeslot */
+ private Map<String, Long> earliestTimeslots = new ConcurrentHashMap<>();
+
+ /** DataPartitionTables collected from DataNodes: dataNodeId -> DataPartitionTable */
+ private Map<Integer, DataPartitionTable> dataPartitionTables = new ConcurrentHashMap<>();
+
+ private Set<String> lostDataPartitionsOfDatabases = new HashSet<>();
+
+ /** Final merged DataPartitionTable */
+ private DataPartitionTable finalDataPartitionTable;
+
+ private static Set<TDataNodeConfiguration> skipDataNodes = new HashSet<>();
+ private static Set<TDataNodeConfiguration> failedDataNodes = new HashSet<>();
+
+ private static ScheduledExecutorService heartBeatExecutor;
+
+ // ============Need serialize END=============/
+
+ public DataPartitionTableIntegrityCheckProcedure() {
+ super();
+ }
+
+ @Override
+ protected Flow executeFromState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws InterruptedException {
+ try {
+ // Ensure to get the real-time DataNodes in the current cluster at every step
+ dataNodeManager = env.getConfigManager().getNodeManager();
+ allDataNodes = dataNodeManager.getRegisteredDataNodes();
+
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ failedDataNodes = new HashSet<>();
+ return collectEarliestTimeslots();
+ case ANALYZE_MISSING_PARTITIONS:
+ lostDataPartitionsOfDatabases = new HashSet<>();
+ return analyzeMissingPartitions(env);
+ case REQUEST_PARTITION_TABLES:
+ heartBeatExecutor = Executors.newScheduledThreadPool(1);
+ return requestPartitionTables(env);
+ case MERGE_PARTITION_TABLES:
+ return mergePartitionTables(env);
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ return writePartitionTableToRaft(env);
+ default:
+ throw new ProcedureException("Unknown state: " + state);
+ }
+ } catch (Exception e) {
+ LOG.error("Error executing state {}: {}", state, e.getMessage(), e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ protected void rollbackState(
+ final ConfigNodeProcedureEnv env, final DataPartitionTableIntegrityCheckProcedureState state)
+ throws IOException, InterruptedException, ProcedureException {
+ switch (state) {
+ case COLLECT_EARLIEST_TIMESLOTS:
+ case ANALYZE_MISSING_PARTITIONS:
+ case REQUEST_PARTITION_TABLES:
+ case MERGE_PARTITION_TABLES:
+ case WRITE_PARTITION_TABLE_TO_RAFT:
+ // Cleanup resources
+ earliestTimeslots.clear();
+ dataPartitionTables.clear();
+ allDataNodes.clear();
+ finalDataPartitionTable = null;
+ break;
+ default:
+ throw new ProcedureException("Unknown state for rollback: " + state);
+ }
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getState(final int stateId) {
+ return DataPartitionTableIntegrityCheckProcedureState.values()[stateId];
+ }
+
+ @Override
+ protected int getStateId(final DataPartitionTableIntegrityCheckProcedureState state) {
+ return state.ordinal();
+ }
+
+ @Override
+ protected DataPartitionTableIntegrityCheckProcedureState getInitialState() {
+ skipDataNodes = new HashSet<>();
+ failedDataNodes = new HashSet<>();
+ return DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS;
+ }
+
+ /**
+ * Collect earliest timeslot information from all DataNodes. Each DataNode returns a Map<String,
+ * Long> where key is database name and value is the earliest timeslot id.
+ */
+ private Flow collectEarliestTimeslots() {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Collecting earliest timeslots from all DataNodes...");
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to collect earliest timeslots, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Collect earliest timeslots from all DataNodes
+ allDataNodes.removeAll(skipDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ try {
+ TGetEarliestTimeslotsResp resp =
+ (TGetEarliestTimeslotsResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.COLLECT_EARLIEST_TIMESLOTS,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to collected earliest timeslots from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ Map<String, Long> nodeTimeslots = resp.getDatabaseToEarliestTimeslot();
+
+ // Merge with existing timeslots (take minimum)
+ for (Map.Entry<String, Long> entry : nodeTimeslots.entrySet()) {
+ earliestTimeslots.merge(entry.getKey(), entry.getValue(), Math::min);
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Collected earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ nodeTimeslots);
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Failed to collect earliest timeslots from the DataNode[id={}]: {}",
+ dataNode.getLocation().getDataNodeId(),
+ e.getMessage(),
+ e);
+ failedDataNodes.add(dataNode);
+ }
+ }
+
+ if (LOG.isDebugEnabled()) {
+ LOG.info(
+ "Collected earliest timeslots from {} DataNodes: {}, the number of successful DataNodes is {}",
+ allDataNodes.size(),
+ earliestTimeslots,
+ allDataNodes.size() - failedDataNodes.size());
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ } else {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.ANALYZE_MISSING_PARTITIONS);
+ }
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /**
+ * Analyze which data partitions are missing based on earliest timeslots. Identify data partitions
+ * of databases need to be repaired.
+ */
+ private Flow analyzeMissingPartitions(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Analyzing missing data partitions...");
+ }
+
+ if (earliestTimeslots.isEmpty()) {
+ LOG.error(
+ "No missing data partitions detected, nothing needs to be repaired, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ // Find all databases that have lost data partition tables
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ String database = entry.getKey();
+ long earliestTimeslot = entry.getValue();
+
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTable = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTable == null
+ || localDataPartitionTable.isEmpty()
+ || localDataPartitionTable.get(database) == null
+ || localDataPartitionTable.get(database).isEmpty()) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, and this issue needs to be repaired",
+ database);
+ continue;
+ }
+
+ Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionMap = localDataPartitionTable.get(database);
+ for (Map.Entry<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>
+ seriesPartitionEntry : seriesPartitionMap.entrySet()) {
+ Map<TTimePartitionSlot, List<TConsensusGroupId>> tTimePartitionSlotListMap =
+ seriesPartitionEntry.getValue();
+ tTimePartitionSlotListMap
+ .keySet()
+ .forEach(
+ slot -> {
+ if (!TimePartitionUtils.satisfyPartitionId(
+ slot.getStartTime(), earliestTimeslot)) {
+ lostDataPartitionsOfDatabases.add(database);
+ LOG.warn(
+ "Database {} has lost timeslot {} in its data table partition, and this issue needs to be repaired",
+ database,
+ earliestTimeslot);
+ }
+ });
+ }
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.info("No databases have lost data partitions, terminating procedure");
+ return Flow.NO_MORE_STATE;
+ }
+
+ LOG.info(
+ "Identified {} databases have lost data partitions, will request DataPartitionTable generation from {} DataNodes",
+ lostDataPartitionsOfDatabases.size(),
+ allDataNodes.size() - failedDataNodes.size());
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.REQUEST_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ private Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ getLocalDataPartitionTable(final ConfigNodeProcedureEnv env, final String database) {
+ Map<String, Map<TSeriesPartitionSlot, TConsensusGroupId>> schemaPartitionTable =
+ env.getConfigManager()
+ .getSchemaPartition(Collections.singletonMap(database, Collections.emptyList()))
+ .getSchemaPartitionTable();
+
+ // Construct request for getting data partition
+ final Map<String, Map<TSeriesPartitionSlot, TTimeSlotList>> partitionSlotsMap = new HashMap<>();
+ schemaPartitionTable.forEach(
+ (key, value) -> {
+ Map<TSeriesPartitionSlot, TTimeSlotList> slotListMap = new HashMap<>();
+ value
+ .keySet()
+ .forEach(
+ slot ->
+ slotListMap.put(
+ slot, new TTimeSlotList(Collections.emptyList(), true, true)));
+ partitionSlotsMap.put(key, slotListMap);
+ });
+ final GetDataPartitionPlan getDataPartitionPlan = new GetDataPartitionPlan(partitionSlotsMap);
+ return env.getConfigManager().getDataPartition(getDataPartitionPlan).getDataPartitionTable();
+ }
+
+ /**
+ * Request DataPartitionTable generation from target DataNodes. Each DataNode scans its tsfile
+ * resources and generates a DataPartitionTable.
+ */
+ private Flow requestPartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(
+ "Requesting DataPartitionTable generation from {} DataNodes...", allDataNodes.size());
+ }
+
+ if (allDataNodes.isEmpty()) {
+ LOG.error(
+ "No DataNodes registered, no way to requested DataPartitionTable generation, terminating procedure");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ ScheduledExecutorUtil.safelyScheduleAtFixedRate(
+ heartBeatExecutor,
+ this::checkPartitionTableGenerationStatus,
+ 0,
+ HEART_BEAT_REQUEST_RATE,
+ TimeUnit.MILLISECONDS);
+
+ allDataNodes.removeAll(skipDataNodes);
+ allDataNodes.removeAll(failedDataNodes);
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableReq req = new TGenerateDataPartitionTableReq();
+ req.setDatabases(lostDataPartitionsOfDatabases);
+ TGenerateDataPartitionTableResp resp =
+ (TGenerateDataPartitionTableResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ req,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE,
+ MAX_RETRY_COUNT);
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from the DataNode[id={}], response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ resp.getStatus());
+ continue;
+ }
+
+ byte[] bytes = resp.getDataPartitionTable();
+ DataPartitionTable dataPartitionTable = new DataPartitionTable();
+ dataPartitionTable.deserialize(ByteBuffer.wrap(bytes));
+ dataPartitionTables.put(dataNodeId, dataPartitionTable);
+ } catch (Exception e) {
+ failedDataNodes.add(dataNode);
+ LOG.error(
+ "Failed to request DataPartitionTable generation from DataNode[id={}]: {}",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ }
+ }
+ }
+
+ if (failedDataNodes.size() == allDataNodes.size()
+ && new HashSet<>(allDataNodes).containsAll(failedDataNodes)) {
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.MERGE_PARTITION_TABLES);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Check completion status of DataPartitionTable generation tasks. */
+ private void checkPartitionTableGenerationStatus() {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Checking DataPartitionTable generation completion status...");
+ }
+
+ int completeCount = 0;
+ for (TDataNodeConfiguration dataNode : allDataNodes) {
+ int dataNodeId = dataNode.getLocation().getDataNodeId();
+
+ if (!dataPartitionTables.containsKey(dataNodeId)) {
+ try {
+ TGenerateDataPartitionTableHeartbeatResp resp =
+ (TGenerateDataPartitionTableHeartbeatResp)
+ SyncDataNodeClientPool.getInstance()
+ .sendSyncRequestToDataNodeWithGivenRetry(
+ dataNode.getLocation().getInternalEndPoint(),
+ null,
+ CnToDnSyncRequestType.GENERATE_DATA_PARTITION_TABLE_HEART_BEAT,
+ MAX_RETRY_COUNT);
+ DataPartitionTableGeneratorState state =
+ DataPartitionTableGeneratorState.getStateByCode(resp.getErrorCode());
+
+ if (resp.getStatus().getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.error(
+ "Failed to request DataPartitionTable generation heart beat from the DataNode[id={}], state is {}, response status is {}",
+ dataNode.getLocation().getDataNodeId(),
+ state,
+ resp.getStatus());
+ continue;
+ }
+ switch (state) {
+ case SUCCESS:
+ LOG.info(
+ "DataNode {} completed DataPartitionTable generation, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ case IN_PROGRESS:
+ LOG.info("DataNode {} still generating DataPartitionTable", dataNodeId);
+ break;
+ case FAILED:
+ LOG.error(
+ "DataNode {} failed to generate DataPartitionTable, terminating heart beat",
+ dataNodeId);
+ completeCount++;
+ break;
+ default:
+ LOG.error(
+ "DataNode {} returned unknown error code: {}", dataNodeId, resp.getErrorCode());
+ break;
+ }
+ } catch (Exception e) {
+ LOG.error(
+ "Error checking DataPartitionTable status from DataNode {}: {}, terminating heart beat",
+ dataNodeId,
+ e.getMessage(),
+ e);
+ completeCount++;
+ }
+ } else {
+ completeCount++;
+ }
+ }
+
+ if (completeCount >= allDataNodes.size()) {
+ heartBeatExecutor.shutdown();
+ }
+ }
+
+ /** Merge DataPartitionTables from all DataNodes into a final table. */
+ private Flow mergePartitionTables(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Merging DataPartitionTables from {} DataNodes...", dataPartitionTables.size());
+ }
+
+ if (dataPartitionTables.isEmpty()) {
+ LOG.error("No DataPartitionTables to merge, dataPartitionTables is empty");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ Map<TSeriesPartitionSlot, SeriesPartitionTable> finalDataPartitionMap = new HashMap<>();
+
+ for (String database : lostDataPartitionsOfDatabases) {
+ // Get current DataPartitionTable from ConfigManager
+ Map<String, Map<TSeriesPartitionSlot, Map<TTimePartitionSlot, List<TConsensusGroupId>>>>
+ localDataPartitionTableMap = getLocalDataPartitionTable(env, database);
+
+ // Check if ConfigNode has a data partition that is associated with the earliestTimeslot
+ if (localDataPartitionTableMap == null
+ || localDataPartitionTableMap.isEmpty()
+ || localDataPartitionTableMap.get(database) == null
+ || localDataPartitionTableMap.get(database).isEmpty()) {
+ LOG.warn(
+ "No data partition table related to database {} was found from the ConfigNode, use data partition table of DataNode directly",
+ database);
+ continue;
+ }
+
+ localDataPartitionTableMap
+ .values()
+ .forEach(
+ map ->
+ map.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTableMap) -> {
+ if (tSeriesPartitionSlot == null
+ || seriesPartitionTableMap == null
+ || seriesPartitionTableMap.isEmpty()) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ tSeriesPartitionSlot,
+ k -> new SeriesPartitionTable(seriesPartitionTableMap));
+ }));
+ }
+
+ if (finalDataPartitionMap.isEmpty()) {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (dnSeriesPartitionSlot == null || dnSeriesPartitionTable == null) {
+ return;
+ }
+ finalDataPartitionMap.computeIfAbsent(
+ dnSeriesPartitionSlot, k -> dnSeriesPartitionTable);
+ });
+ });
+ } else {
+ finalDataPartitionMap.forEach(
+ (tSeriesPartitionSlot, seriesPartitionTable) -> {
+ dataPartitionTables
+ .values()
+ .forEach(
+ dataPartitionTable -> {
+ if (dataPartitionTable == null
+ || dataPartitionTable.getDataPartitionMap() == null
+ || dataPartitionTable.getDataPartitionMap().isEmpty()) {
+ return;
+ }
+ dataPartitionTable
+ .getDataPartitionMap()
+ .forEach(
+ (dnSeriesPartitionSlot, dnSeriesPartitionTable) -> {
+ if (!tSeriesPartitionSlot.equals(dnSeriesPartitionSlot)) {
+ return;
+ }
+
+ if (seriesPartitionTable == null
+ || seriesPartitionTable.getSeriesPartitionMap() == null
+ || seriesPartitionTable.getSeriesPartitionMap().isEmpty()) {
+ finalDataPartitionMap.put(
+ tSeriesPartitionSlot, dnSeriesPartitionTable);
+ }
+
+ // dnDataPartitionTable merged to seriesPartitionTable
+ dnSeriesPartitionTable
+ .getSeriesPartitionMap()
+ .forEach(
+ (k, v) ->
+ v.forEach(
+ tConsensusGroupId -> {
+ if (seriesPartitionTable == null) {
+ return;
+ }
+ seriesPartitionTable.putDataPartition(
+ k, tConsensusGroupId);
+ }));
+ });
+ });
+ });
+ }
+
+ finalDataPartitionTable = new DataPartitionTable(finalDataPartitionMap);
+
+ LOG.info("DataPartitionTable merge completed successfully");
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.WRITE_PARTITION_TABLE_TO_RAFT);
+ return Flow.HAS_MORE_STATE;
+ }
+
+ /** Write the final DataPartitionTable to raft log. */
+ private Flow writePartitionTableToRaft(final ConfigNodeProcedureEnv env) {
+ if (LOG.isDebugEnabled()) {
+ LOG.info("Writing DataPartitionTable to raft log...");
+ }
+
+ if (lostDataPartitionsOfDatabases.isEmpty()) {
+ LOG.error("No database lost data partition table");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No database lost data partition table for raft write"));
+ return getFlow();
+ }
+
+ if (finalDataPartitionTable == null) {
+ LOG.error("No DataPartitionTable to write to raft");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("No DataPartitionTable available for raft write"));
+ return getFlow();
+ }
+
+ int failedCnt = 0;
+ while (failedCnt < MAX_RETRY_COUNT) {
+ try {
+ CreateDataPartitionPlan createPlan = new CreateDataPartitionPlan();
+ Map<String, DataPartitionTable> assignedDataPartition = new HashMap<>();
+ assignedDataPartition.put(
+ lostDataPartitionsOfDatabases.stream().findFirst().get(), finalDataPartitionTable);
+ createPlan.setAssignedDataPartition(assignedDataPartition);
+ TSStatus tsStatus = env.getConfigManager().getConsensusManager().write(createPlan);
+
+ if (tsStatus.getCode() == TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ LOG.info("DataPartitionTable successfully written to raft log");
+ break;
+ } else {
+ LOG.error("Failed to write DataPartitionTable to raft log");
+ setFailure(
+ "DataPartitionTableIntegrityCheckProcedure",
+ new ProcedureException("Failed to write DataPartitionTable to raft log"));
+ }
+ } catch (Exception e) {
+ LOG.error("Error writing DataPartitionTable to raft log", e);
+ setFailure("DataPartitionTableIntegrityCheckProcedure", e);
+ }
+ failedCnt++;
+ }
+
+ return getFlow();
+ }
+
+ private Flow getFlow() {
+ if (!failedDataNodes.isEmpty()) {
+ allDataNodes.removeAll(failedDataNodes);
+ skipDataNodes = new HashSet<>(allDataNodes);
+ setNextState(DataPartitionTableIntegrityCheckProcedureState.COLLECT_EARLIEST_TIMESLOTS);
+ return Flow.HAS_MORE_STATE;
+ } else {
+ skipDataNodes.clear();
+ return Flow.NO_MORE_STATE;
+ }
+ }
+
+ @Override
+ public void serialize(final DataOutputStream stream) throws IOException {
+ super.serialize(stream);
+
+ // Serialize earliestTimeslots
+ stream.writeInt(earliestTimeslots.size());
+ for (Map.Entry<String, Long> entry : earliestTimeslots.entrySet()) {
+ stream.writeUTF(entry.getKey());
+ stream.writeLong(entry.getValue());
+ }
+
+ // Serialize dataPartitionTables count
+ stream.writeInt(dataPartitionTables.size());
+ for (Map.Entry<Integer, DataPartitionTable> entry : dataPartitionTables.entrySet()) {
+ stream.writeInt(entry.getKey());
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ ObjectOutputStream oos = new ObjectOutputStream(baos)) {
+ TTransport transport = new TIOStreamTransport(oos);
+ TBinaryProtocol protocol = new TBinaryProtocol(transport);
+ entry.getValue().serialize(oos, protocol);
+
+ // Write the size and data for byte array after serialize
+ byte[] data = baos.toByteArray();
+ stream.writeInt(data.length);
+ stream.write(data);
+ } catch (IOException | TException e) {
+ LOG.error("{} serialize failed", this.getClass().getSimpleName(), e);
+ throw new IOException("Failed to serialize dataPartitionTables", e);
+ }
+ }
+
+ stream.writeInt(lostDataPartitionsOfDatabases.size());
+ for (String database : lostDataPartitionsOfDatabases) {
+ stream.writeUTF(database);
Review Comment:
ReadWriteIOUtils.writeString() is not exist, ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2958438198
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -3117,4 +3136,350 @@ public TSStatus writeAuditLog(TAuditLogReq req) {
public void handleClientExit() {
// Do nothing
}
+
+ // ====================================================
+ // Data Partition Table Integrity Check Implementation
+ // ====================================================
+
+ private volatile DataPartitionTableGenerator currentGenerator;
+ private volatile CompletableFuture<Void> currentGeneratorFuture;
+ private volatile long currentTaskId = 0;
+
+ @Override
+ public TGetEarliestTimeslotsResp getEarliestTimeslots() {
+ TGetEarliestTimeslotsResp resp = new TGetEarliestTimeslotsResp();
+
+ try {
+ Map<String, Long> earliestTimeslots = new HashMap<>();
+
+ // Get data directories from configuration
+ String[] dataDirs = IoTDBDescriptor.getInstance().getConfig().getDataDirs();
+
+ for (String dataDir : dataDirs) {
+ File dir = new File(dataDir);
+ if (dir.exists() && dir.isDirectory()) {
+ processDataDirectoryForEarliestTimeslots(dir, earliestTimeslots);
+ }
+ }
Review Comment:
Changed yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically repair lost data partitions [iotdb]
Posted by "zerolbsony (via GitHub)" <gi...@apache.org>.
zerolbsony commented on code in PR #17279:
URL: https://github.com/apache/iotdb/pull/17279#discussion_r2957350888
##########
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template:
##########
@@ -742,6 +742,21 @@ failure_detector_phi_acceptable_pause_in_ms=10000
# Datatype: double(percentage)
disk_space_warning_threshold=0.05
+# The number of threads used for parallel scanning in the partition table recovery
+# effectiveMode: restart
+# Datatype: Integer
+partition_table_recover_worker_num=10
+
+# Limit the number of bytes read per second from a file, the unit is MB
Review Comment:
> in what scenario
To scan data dir parallel ago, to scan data region parallel now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org