You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/12/28 09:40:54 UTC
[GitHub] [iotdb] shuwenwei opened a new pull request, #8644: [IOTDB-4674] Reimplement settle by compaction
shuwenwei opened a new pull request, #8644:
URL: https://github.com/apache/iotdb/pull/8644
## Description
Reimplement settle by compaction. Add a online tool which wraps some TsFiles to an inner space compaction task to implement settle.
doc: https://apache-iotdb.feishu.cn/docx/ByG6d6hllorhbdxxoqscLamEnsb
### Usage
```
#MacOs or Linux
./settle-tsfile.sh -h [host] -p [port] -f [filePaths]
#Windows
.\settle-tsfile.bat -h [host] -p [port] -f [filePaths]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061103968
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
Review Comment:
You can also define a class to represent the validation result if the validate method need return multiple values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061104604
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
+ dataRegionId = fileDataRegionId;
+ dataRegion = StorageEngine.getInstance().getDataRegion(new DataRegionId(dataRegionId));
+ } else if (dataRegionId != fileDataRegionId) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "DataRegion of files is not consistent.");
+ }
+
+ String sgOfFile = TsFileUtils.getStorageGroup(tsFile);
+ if (storageGroupName == null) {
+ storageGroupName = sgOfFile;
+ } else if (!storageGroupName.equals(sgOfFile)) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "StorageGroup of files is not consistent.");
+ }
+
+ long timePartitionOfFile = TsFileUtils.getTimePartition(tsFile);
+ if (timePartitionId == null) {
+ timePartitionId = timePartitionOfFile;
+ } else if (timePartitionId != timePartitionOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "TimePartition of files is not consistent.");
+ }
+
+ String fileName = tsFile.getName();
+ TsFileNameGenerator.TsFileName tsFileName;
+ try {
+ tsFileName = TsFileNameGenerator.getTsFileName(fileName);
+ } catch (IOException e) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "Meet error when parsing TsFileName.");
+ }
+
+ int levelOfFile = tsFileName.getInnerCompactionCnt();
+ if (level == null) {
+ level = levelOfFile;
+ } else if (level != levelOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Level of files is not consistent.");
+ }
+
+ if (TsFileUtils.isSequence(tsFile)) {
+ hasSeqFile = true;
+ } else {
+ hasUnSeqFile = true;
+ }
+ tsFileNames.add(fileName);
+ if (hasSeqFile && hasUnSeqFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "At least one mods file should be selected");
+ }
+ if (dataRegion == null) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "DataRegion not exist");
+ }
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ if (!tsFileManager.isAllowCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in this DataRegion is not allowed.");
+ }
+
+ // try to get a continuous TsFileResource list with input TsFile names
+ List<TsFileResource> tsFileResources =
+ getContinuousTsFileResourcesByFileNames(
+ tsFileManager, timePartitionId, hasSeqFile, tsFileNames);
+ if (tsFileResources.size() != tsFileNames.size()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Could not find enough satisfied TsFile");
Review Comment:
Optimize the description. What we want to tell users is that selected tsfiles are not continuous
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061104334
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
+ dataRegionId = fileDataRegionId;
+ dataRegion = StorageEngine.getInstance().getDataRegion(new DataRegionId(dataRegionId));
+ } else if (dataRegionId != fileDataRegionId) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "DataRegion of files is not consistent.");
+ }
+
+ String sgOfFile = TsFileUtils.getStorageGroup(tsFile);
+ if (storageGroupName == null) {
+ storageGroupName = sgOfFile;
+ } else if (!storageGroupName.equals(sgOfFile)) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "StorageGroup of files is not consistent.");
+ }
+
+ long timePartitionOfFile = TsFileUtils.getTimePartition(tsFile);
+ if (timePartitionId == null) {
+ timePartitionId = timePartitionOfFile;
+ } else if (timePartitionId != timePartitionOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "TimePartition of files is not consistent.");
+ }
+
+ String fileName = tsFile.getName();
+ TsFileNameGenerator.TsFileName tsFileName;
+ try {
+ tsFileName = TsFileNameGenerator.getTsFileName(fileName);
+ } catch (IOException e) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "Meet error when parsing TsFileName.");
+ }
+
+ int levelOfFile = tsFileName.getInnerCompactionCnt();
+ if (level == null) {
+ level = levelOfFile;
+ } else if (level != levelOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Level of files is not consistent.");
+ }
+
+ if (TsFileUtils.isSequence(tsFile)) {
+ hasSeqFile = true;
+ } else {
+ hasUnSeqFile = true;
+ }
+ tsFileNames.add(fileName);
+ if (hasSeqFile && hasUnSeqFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "At least one mods file should be selected");
Review Comment:
Change the description because it may confuse the users. What we want to tell users is that every selected TsFile does not contains the mod file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061105028
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
+ dataRegionId = fileDataRegionId;
+ dataRegion = StorageEngine.getInstance().getDataRegion(new DataRegionId(dataRegionId));
+ } else if (dataRegionId != fileDataRegionId) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "DataRegion of files is not consistent.");
+ }
+
+ String sgOfFile = TsFileUtils.getStorageGroup(tsFile);
+ if (storageGroupName == null) {
+ storageGroupName = sgOfFile;
+ } else if (!storageGroupName.equals(sgOfFile)) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "StorageGroup of files is not consistent.");
+ }
+
+ long timePartitionOfFile = TsFileUtils.getTimePartition(tsFile);
+ if (timePartitionId == null) {
+ timePartitionId = timePartitionOfFile;
+ } else if (timePartitionId != timePartitionOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "TimePartition of files is not consistent.");
+ }
+
+ String fileName = tsFile.getName();
+ TsFileNameGenerator.TsFileName tsFileName;
+ try {
+ tsFileName = TsFileNameGenerator.getTsFileName(fileName);
+ } catch (IOException e) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "Meet error when parsing TsFileName.");
+ }
+
+ int levelOfFile = tsFileName.getInnerCompactionCnt();
+ if (level == null) {
+ level = levelOfFile;
+ } else if (level != levelOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Level of files is not consistent.");
+ }
+
+ if (TsFileUtils.isSequence(tsFile)) {
+ hasSeqFile = true;
+ } else {
+ hasUnSeqFile = true;
+ }
+ tsFileNames.add(fileName);
+ if (hasSeqFile && hasUnSeqFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "At least one mods file should be selected");
+ }
+ if (dataRegion == null) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "DataRegion not exist");
+ }
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ if (!tsFileManager.isAllowCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in this DataRegion is not allowed.");
+ }
+
+ // try to get a continuous TsFileResource list with input TsFile names
+ List<TsFileResource> tsFileResources =
+ getContinuousTsFileResourcesByFileNames(
+ tsFileManager, timePartitionId, hasSeqFile, tsFileNames);
+ if (tsFileResources.size() != tsFileNames.size()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Could not find enough satisfied TsFile");
+ }
+
+ IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ if (hasSeqFile && !config.isEnableSeqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Seq Space is not enabled");
+ }
+ if (hasUnSeqFile && !config.isEnableUnseqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Unseq Space is not enabled");
+ }
+ if (tsFileResources.size() > config.getMaxInnerCompactionCandidateFileNum()) {
+ return RpcUtils.getStatus(TSStatusCode.UNSUPPORTED_OPERATION, "File nums is too much.");
+ }
+ if (testMode) {
+ return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+ }
+
+ AbstractCompactionTask task =
+ new InnerSpaceCompactionTask(
+ timePartitionId,
+ tsFileManager,
+ tsFileResources,
+ hasSeqFile,
+ config.getInnerSeqCompactionPerformer().createInstance(),
+ CompactionTaskManager.currentTaskNum,
+ tsFileManager.getNextCompactionTaskId());
+ try {
+ CompactionTaskManager.getInstance().addTaskToWaitingQueue(task);
+ } catch (InterruptedException e) {
+ logger.error(
Review Comment:
Error msg should include the detailed cause from `e`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061107066
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
+ dataRegionId = fileDataRegionId;
+ dataRegion = StorageEngine.getInstance().getDataRegion(new DataRegionId(dataRegionId));
+ } else if (dataRegionId != fileDataRegionId) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "DataRegion of files is not consistent.");
+ }
+
+ String sgOfFile = TsFileUtils.getStorageGroup(tsFile);
+ if (storageGroupName == null) {
+ storageGroupName = sgOfFile;
+ } else if (!storageGroupName.equals(sgOfFile)) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "StorageGroup of files is not consistent.");
+ }
+
+ long timePartitionOfFile = TsFileUtils.getTimePartition(tsFile);
+ if (timePartitionId == null) {
+ timePartitionId = timePartitionOfFile;
+ } else if (timePartitionId != timePartitionOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "TimePartition of files is not consistent.");
+ }
+
+ String fileName = tsFile.getName();
+ TsFileNameGenerator.TsFileName tsFileName;
+ try {
+ tsFileName = TsFileNameGenerator.getTsFileName(fileName);
+ } catch (IOException e) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "Meet error when parsing TsFileName.");
+ }
+
+ int levelOfFile = tsFileName.getInnerCompactionCnt();
+ if (level == null) {
+ level = levelOfFile;
+ } else if (level != levelOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Level of files is not consistent.");
+ }
+
+ if (TsFileUtils.isSequence(tsFile)) {
+ hasSeqFile = true;
+ } else {
+ hasUnSeqFile = true;
+ }
+ tsFileNames.add(fileName);
+ if (hasSeqFile && hasUnSeqFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "At least one mods file should be selected");
+ }
+ if (dataRegion == null) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "DataRegion not exist");
+ }
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ if (!tsFileManager.isAllowCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in this DataRegion is not allowed.");
+ }
+
+ // try to get a continuous TsFileResource list with input TsFile names
+ List<TsFileResource> tsFileResources =
+ getContinuousTsFileResourcesByFileNames(
+ tsFileManager, timePartitionId, hasSeqFile, tsFileNames);
+ if (tsFileResources.size() != tsFileNames.size()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Could not find enough satisfied TsFile");
+ }
+
+ IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ if (hasSeqFile && !config.isEnableSeqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Seq Space is not enabled");
+ }
+ if (hasUnSeqFile && !config.isEnableUnseqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Unseq Space is not enabled");
+ }
+ if (tsFileResources.size() > config.getMaxInnerCompactionCandidateFileNum()) {
+ return RpcUtils.getStatus(TSStatusCode.UNSUPPORTED_OPERATION, "File nums is too much.");
+ }
+ if (testMode) {
+ return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+ }
+
+ AbstractCompactionTask task =
+ new InnerSpaceCompactionTask(
+ timePartitionId,
+ tsFileManager,
+ tsFileResources,
+ hasSeqFile,
+ config.getInnerSeqCompactionPerformer().createInstance(),
+ CompactionTaskManager.currentTaskNum,
+ tsFileManager.getNextCompactionTaskId());
+ try {
+ CompactionTaskManager.getInstance().addTaskToWaitingQueue(task);
+ } catch (InterruptedException e) {
+ logger.error(
+ "meet error when adding task-{} to compaction waiting queue", task.getSerialId());
+ return RpcUtils.getStatus(
+ TSStatusCode.COMPACTION_ERROR, "meet error when submit settle task.");
+ }
+ return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+ }
+
+ private List<TsFileResource> getContinuousTsFileResourcesByFileNames(
Review Comment:
Actually what you want to do are two things:
1. validate whether the selected files are continuous
2. If all the validation are passed, get the corresponding TsFileResource.
Do not merge these two functionalities into one method because it will make the code un-readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061102649
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
Review Comment:
Your implementation is definitely correct but it is better to make the method more concise by split the functionalities into several method such as `calculate target value`, `validation` and `submit task`.
For the `calculate target value` part, we can define a private static class to represent the target values including `dataRegionId`, `storageGroupName`, `level` and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061100536
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
Review Comment:
give more detailed error message including `which file is not existed`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061106008
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
+ dataRegionId = fileDataRegionId;
+ dataRegion = StorageEngine.getInstance().getDataRegion(new DataRegionId(dataRegionId));
+ } else if (dataRegionId != fileDataRegionId) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "DataRegion of files is not consistent.");
+ }
+
+ String sgOfFile = TsFileUtils.getStorageGroup(tsFile);
+ if (storageGroupName == null) {
+ storageGroupName = sgOfFile;
+ } else if (!storageGroupName.equals(sgOfFile)) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "StorageGroup of files is not consistent.");
+ }
+
+ long timePartitionOfFile = TsFileUtils.getTimePartition(tsFile);
+ if (timePartitionId == null) {
+ timePartitionId = timePartitionOfFile;
+ } else if (timePartitionId != timePartitionOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "TimePartition of files is not consistent.");
+ }
+
+ String fileName = tsFile.getName();
+ TsFileNameGenerator.TsFileName tsFileName;
+ try {
+ tsFileName = TsFileNameGenerator.getTsFileName(fileName);
+ } catch (IOException e) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "Meet error when parsing TsFileName.");
+ }
+
+ int levelOfFile = tsFileName.getInnerCompactionCnt();
+ if (level == null) {
+ level = levelOfFile;
+ } else if (level != levelOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Level of files is not consistent.");
+ }
+
+ if (TsFileUtils.isSequence(tsFile)) {
+ hasSeqFile = true;
+ } else {
+ hasUnSeqFile = true;
+ }
+ tsFileNames.add(fileName);
+ if (hasSeqFile && hasUnSeqFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "At least one mods file should be selected");
+ }
+ if (dataRegion == null) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "DataRegion not exist");
+ }
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ if (!tsFileManager.isAllowCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in this DataRegion is not allowed.");
+ }
+
+ // try to get a continuous TsFileResource list with input TsFile names
+ List<TsFileResource> tsFileResources =
+ getContinuousTsFileResourcesByFileNames(
+ tsFileManager, timePartitionId, hasSeqFile, tsFileNames);
+ if (tsFileResources.size() != tsFileNames.size()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Could not find enough satisfied TsFile");
+ }
+
+ IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ if (hasSeqFile && !config.isEnableSeqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Seq Space is not enabled");
+ }
+ if (hasUnSeqFile && !config.isEnableUnseqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Unseq Space is not enabled");
+ }
+ if (tsFileResources.size() > config.getMaxInnerCompactionCandidateFileNum()) {
Review Comment:
We can move this validation before the continuous file validation because this check is more simple to do, which will reduce the resource usage in some scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1063981554
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ SettleRequestContext context = new SettleRequestContext(paths);
+ TSStatus validationResult = context.validate();
+ if (validationResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ return validationResult;
+ }
+ if (testMode) {
+ return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+ }
+ List<TsFileResource> selectedTsFileResources = context.getTsFileResourcesByFileNames();
+ return context.submitCompactionTask(selectedTsFileResources);
+ }
+
+ private static class SettleRequestContext {
+ private ConsistentSettleInfo consistentSettleInfo;
+
+ private boolean hasSeqFiles;
+ private boolean hasUnseqFiles;
+ private boolean hasModsFiles;
+ private List<String> paths;
+ private Set<String> tsFileNames;
+ private TsFileResourceList allTsFileResourceList;
+
+ private TsFileManager tsFileManager;
+ private IoTDBConfig config;
+
+ private SettleRequestContext(List<String> paths) {
+ this.paths = paths;
+ this.tsFileNames = new HashSet<>();
+ this.config = IoTDBDescriptor.getInstance().getConfig();
+ }
+
+ private TSStatus validate() {
+ if (paths == null || paths.size() == 0) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "The files to settle is not offered.");
+ }
+
+ int maxInnerCompactionCandidateFileNum = config.getMaxInnerCompactionCandidateFileNum();
+ if (paths.size() > maxInnerCompactionCandidateFileNum) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION,
+ "Too many files offered, the limited count of system config is "
+ + maxInnerCompactionCandidateFileNum
+ + ", the input file count is "
+ + tsFileNames.size());
+ }
+
+ TSStatus validationResult;
+ Pair<ConsistentSettleInfo, TSStatus> consistentSettleInfoStatusPair;
+
+ for (String path : paths) {
+ File currentTsFile = new File(path);
+ if (!currentTsFile.exists()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.PATH_NOT_EXIST, "The specified file does not exist in " + path);
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFiles |= modsFile.exists();
+
+ consistentSettleInfoStatusPair = calculateConsistentInfo(currentTsFile);
+ ConsistentSettleInfo currentInfo = consistentSettleInfoStatusPair.left;
+ if (this.consistentSettleInfo == null) {
+ this.consistentSettleInfo = currentInfo;
+ }
+ validationResult = consistentSettleInfoStatusPair.right;
+ if (!isSuccess(validationResult)) {
+ return validationResult;
+ }
+
+ validationResult = consistentSettleInfo.checkConsistency(currentInfo);
+ if (!isSuccess(validationResult)) {
+ return validationResult;
+ }
+
+ if (TsFileUtils.isSequence(currentTsFile)) {
+ hasSeqFiles = true;
+ } else {
+ hasUnseqFiles = true;
+ }
+ tsFileNames.add(currentTsFile.getName());
+ if (hasSeqFiles && hasUnseqFiles) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFiles) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER,
+ "Every selected TsFile does not contains the mods file.");
+ }
+ DataRegion dataRegion =
+ StorageEngine.getInstance()
+ .getDataRegion(new DataRegionId(consistentSettleInfo.dataRegionId));
+ if (dataRegion == null) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "DataRegion not exist");
+ }
+ tsFileManager = dataRegion.getTsFileManager();
+
+ validationResult = checkCompactionConfigs();
+ if (!isSuccess(validationResult)) {
+ return validationResult;
+ }
+
+ if (hasSeqFiles) {
+ allTsFileResourceList =
+ tsFileManager.getSequenceListByTimePartition(consistentSettleInfo.timePartitionId);
+ } else {
+ allTsFileResourceList =
+ tsFileManager.getUnsequenceListByTimePartition(consistentSettleInfo.timePartitionId);
+ }
+
+ return validateTsFileResources();
+ }
+
+ private Pair<ConsistentSettleInfo, TSStatus> calculateConsistentInfo(File tsFile) {
Review Comment:
We need to avoid using `Pair` because it will add the complexity of codes.
We can put the `status` as a variable of ConsistentSettleInfo.
What you want to do here is check whether the `info` can be totally got from given TsFile path or not. It not, the whole procedure should terminated.
So we can add a new variable such as `isValid` into `ConsistentSettleInfo` to represent whether the Info is ok or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061101432
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
Review Comment:
We can give them a more detailed var name such as `targetDataRegionId`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1063981664
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ SettleRequestContext context = new SettleRequestContext(paths);
+ TSStatus validationResult = context.validate();
+ if (validationResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ return validationResult;
+ }
+ if (testMode) {
+ return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+ }
+ List<TsFileResource> selectedTsFileResources = context.getTsFileResourcesByFileNames();
+ return context.submitCompactionTask(selectedTsFileResources);
+ }
+
+ private static class SettleRequestContext {
+ private ConsistentSettleInfo consistentSettleInfo;
+
+ private boolean hasSeqFiles;
+ private boolean hasUnseqFiles;
+ private boolean hasModsFiles;
+ private List<String> paths;
+ private Set<String> tsFileNames;
+ private TsFileResourceList allTsFileResourceList;
+
+ private TsFileManager tsFileManager;
+ private IoTDBConfig config;
+
+ private SettleRequestContext(List<String> paths) {
+ this.paths = paths;
+ this.tsFileNames = new HashSet<>();
+ this.config = IoTDBDescriptor.getInstance().getConfig();
+ }
+
+ private TSStatus validate() {
+ if (paths == null || paths.size() == 0) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "The files to settle is not offered.");
+ }
+
+ int maxInnerCompactionCandidateFileNum = config.getMaxInnerCompactionCandidateFileNum();
+ if (paths.size() > maxInnerCompactionCandidateFileNum) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION,
+ "Too many files offered, the limited count of system config is "
+ + maxInnerCompactionCandidateFileNum
+ + ", the input file count is "
+ + tsFileNames.size());
+ }
+
+ TSStatus validationResult;
+ Pair<ConsistentSettleInfo, TSStatus> consistentSettleInfoStatusPair;
+
+ for (String path : paths) {
+ File currentTsFile = new File(path);
+ if (!currentTsFile.exists()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.PATH_NOT_EXIST, "The specified file does not exist in " + path);
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFiles |= modsFile.exists();
+
+ consistentSettleInfoStatusPair = calculateConsistentInfo(currentTsFile);
+ ConsistentSettleInfo currentInfo = consistentSettleInfoStatusPair.left;
+ if (this.consistentSettleInfo == null) {
+ this.consistentSettleInfo = currentInfo;
+ }
+ validationResult = consistentSettleInfoStatusPair.right;
+ if (!isSuccess(validationResult)) {
Review Comment:
This judgement can be moved before line `134`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1059290615
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+ private static final String MODS_FILE_SUFFIX = ".mods";
Review Comment:
can refer `ModificationFile.FILE_SUFFIX`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1063981610
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,321 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.Pair;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ SettleRequestContext context = new SettleRequestContext(paths);
+ TSStatus validationResult = context.validate();
+ if (validationResult.getCode() != TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ return validationResult;
+ }
+ if (testMode) {
+ return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+ }
+ List<TsFileResource> selectedTsFileResources = context.getTsFileResourcesByFileNames();
+ return context.submitCompactionTask(selectedTsFileResources);
+ }
+
+ private static class SettleRequestContext {
+ private ConsistentSettleInfo consistentSettleInfo;
Review Comment:
Rename to `targetConsistentSettleInfo`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr merged pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr merged PR #8644:
URL: https://github.com/apache/iotdb/pull/8644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061098062
##########
server/src/main/java/org/apache/iotdb/db/tools/settle/TsFileSettleByCompactionTool.java:
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.tools.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.mpp.rpc.thrift.IDataNodeRPCService;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.CommandLineParser;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.Option;
+import org.apache.commons.cli.Options;
+import org.apache.commons.cli.ParseException;
+import org.apache.thrift.TException;
+import org.apache.thrift.protocol.TBinaryProtocol;
+import org.apache.thrift.protocol.TProtocol;
+import org.apache.thrift.transport.TSocket;
+import org.apache.thrift.transport.TTransport;
+import org.apache.thrift.transport.layered.TFramedTransport;
+
+import java.util.Arrays;
+
+public class TsFileSettleByCompactionTool {
+
+ private static final String HOST_ARGS = "h";
+ private static final String HOST_NAME = "host";
+
+ private static final String PORT_ARGS = "p";
+ private static final String PORT_NAME = "port";
+
+ private static final String FILE_PATH_ARGS = "f";
+ private static final String FILE_PATH_NAME = "file paths";
+
+ public static void main(String[] args) throws TException {
+ String hostValue = "127.0.0.1";
Review Comment:
Suggest to define `static final string` as the default value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061104857
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
+ dataRegionId = fileDataRegionId;
+ dataRegion = StorageEngine.getInstance().getDataRegion(new DataRegionId(dataRegionId));
+ } else if (dataRegionId != fileDataRegionId) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "DataRegion of files is not consistent.");
+ }
+
+ String sgOfFile = TsFileUtils.getStorageGroup(tsFile);
+ if (storageGroupName == null) {
+ storageGroupName = sgOfFile;
+ } else if (!storageGroupName.equals(sgOfFile)) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "StorageGroup of files is not consistent.");
+ }
+
+ long timePartitionOfFile = TsFileUtils.getTimePartition(tsFile);
+ if (timePartitionId == null) {
+ timePartitionId = timePartitionOfFile;
+ } else if (timePartitionId != timePartitionOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PATH, "TimePartition of files is not consistent.");
+ }
+
+ String fileName = tsFile.getName();
+ TsFileNameGenerator.TsFileName tsFileName;
+ try {
+ tsFileName = TsFileNameGenerator.getTsFileName(fileName);
+ } catch (IOException e) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "Meet error when parsing TsFileName.");
+ }
+
+ int levelOfFile = tsFileName.getInnerCompactionCnt();
+ if (level == null) {
+ level = levelOfFile;
+ } else if (level != levelOfFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Level of files is not consistent.");
+ }
+
+ if (TsFileUtils.isSequence(tsFile)) {
+ hasSeqFile = true;
+ } else {
+ hasUnSeqFile = true;
+ }
+ tsFileNames.add(fileName);
+ if (hasSeqFile && hasUnSeqFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Settle by cross compaction is not allowed.");
+ }
+ }
+
+ if (!hasModsFile) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "At least one mods file should be selected");
+ }
+ if (dataRegion == null) {
+ return RpcUtils.getStatus(TSStatusCode.ILLEGAL_PATH, "DataRegion not exist");
+ }
+ TsFileManager tsFileManager = dataRegion.getTsFileManager();
+ if (!tsFileManager.isAllowCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in this DataRegion is not allowed.");
+ }
+
+ // try to get a continuous TsFileResource list with input TsFile names
+ List<TsFileResource> tsFileResources =
+ getContinuousTsFileResourcesByFileNames(
+ tsFileManager, timePartitionId, hasSeqFile, tsFileNames);
+ if (tsFileResources.size() != tsFileNames.size()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.ILLEGAL_PARAMETER, "Could not find enough satisfied TsFile");
+ }
+
+ IoTDBConfig config = IoTDBDescriptor.getInstance().getConfig();
+ if (hasSeqFile && !config.isEnableSeqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Seq Space is not enabled");
+ }
+ if (hasUnSeqFile && !config.isEnableUnseqSpaceCompaction()) {
+ return RpcUtils.getStatus(
+ TSStatusCode.UNSUPPORTED_OPERATION, "Compaction in Unseq Space is not enabled");
+ }
+ if (tsFileResources.size() > config.getMaxInnerCompactionCandidateFileNum()) {
+ return RpcUtils.getStatus(TSStatusCode.UNSUPPORTED_OPERATION, "File nums is too much.");
Review Comment:
Add more detailed message including current selected file count and limited count
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #8644: [IOTDB-4674] Reimplement settle by compaction
Posted by GitBox <gi...@apache.org>.
xingtanzjr commented on code in PR #8644:
URL: https://github.com/apache/iotdb/pull/8644#discussion_r1061101196
##########
server/src/main/java/org/apache/iotdb/db/engine/settle/SettleRequestHandler.java:
##########
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.settle;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.common.rpc.thrift.TSettleReq;
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.compaction.CompactionTaskManager;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionTask;
+import org.apache.iotdb.db.engine.modification.ModificationFile;
+import org.apache.iotdb.db.engine.storagegroup.DataRegion;
+import org.apache.iotdb.db.engine.storagegroup.TsFileManager;
+import org.apache.iotdb.db.engine.storagegroup.TsFileNameGenerator;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceStatus;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+import org.apache.iotdb.tsfile.utils.TsFileUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+public class SettleRequestHandler {
+ private static final Logger logger = LoggerFactory.getLogger(SettleRequestHandler.class);
+
+ private boolean testMode = false;
+
+ public boolean isTestMode() {
+ return testMode;
+ }
+
+ public void setTestMode(boolean testMode) {
+ this.testMode = testMode;
+ }
+
+ public static SettleRequestHandler getInstance() {
+ return SettleRequestHandlerHolder.INSTANCE;
+ }
+
+ public TSStatus handleSettleRequest(TSettleReq req) {
+ List<String> paths = req.getPaths();
+
+ boolean hasSeqFile = false, hasUnSeqFile = false, hasModsFile = false;
+ Integer dataRegionId = null;
+ DataRegion dataRegion = null;
+ Integer level = null;
+ String storageGroupName = null;
+ Long timePartitionId = null;
+
+ Set<String> tsFileNames = new HashSet<>();
+
+ for (String path : paths) {
+ File tsFile = new File(path);
+ if (!tsFile.exists()) {
+ return RpcUtils.getStatus(TSStatusCode.PATH_NOT_EXIST, "file not exist.");
+ }
+ File modsFile = new File(path + ModificationFile.FILE_SUFFIX);
+ hasModsFile |= modsFile.exists();
+
+ int fileDataRegionId = TsFileUtils.getDataRegionId(tsFile);
+ if (dataRegion == null) {
Review Comment:
Suggest to use first TsFile to get the target value of `dataRegion`, `storageGroupName`, `timePartitionId` and `level` to avoid `if-else` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org