You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2021/06/24 02:44:30 UTC
[GitHub] [iotdb] THUMarkLau opened a new pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
THUMarkLau opened a new pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447
We propose a new compaction strategy and compaction scheduling strategy in [issue 3445](https://github.com/apache/iotdb/issues/3445)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41638054)
Coverage increased (+0.02%) to 68.098% when pulling **05b6e286d6f6e1e3c9757785835b11e2a48204b0 on THUMarkLau:new_compaction** into **72799f33215d5f6291544c354caf58c23a3dd04a on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41829867)
Coverage increased (+0.1%) to 68.3% when pulling **76c46f3bb825f0ef1f900f1befc9947bde1bc588 on THUMarkLau:new_compaction** into **1d0b6ff0893a8ea1e43ad66d1b589770bc898933 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41711963)
Coverage decreased (-0.03%) to 68.188% when pulling **c51ce1c87e9c8a0541dc4ed06ee010fa97bf5a41 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42193347)
Coverage increased (+0.04%) to 67.265% when pulling **117ad54576dd2392f3f019775462218f32640e83 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42270777)
Coverage increased (+0.03%) to 67.263% when pulling **5c091df67ea5bacbcc6410d69486c32afc5b2dd6 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42229505)
Coverage increased (+0.02%) to 67.247% when pulling **c7b01da3f989a9fc4b6d9e42f31bf4da36d83e3b on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42269050)
Coverage increased (+0.03%) to 67.262% when pulling **2e100c034635a8afdeafe2a1d0d9aed02e9d53cc on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681047014
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2124,12 +2050,10 @@ public int countUpgradeFiles() {
/** upgrade all files belongs to this storage group */
public void upgrade() {
for (TsFileResource seqTsFileResource : upgradeSeqFileList) {
- seqTsFileResource.setSeq(true);
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2124,12 +2050,10 @@ public int countUpgradeFiles() {
/** upgrade all files belongs to this storage group */
public void upgrade() {
for (TsFileResource seqTsFileResource : upgradeSeqFileList) {
- seqTsFileResource.setSeq(true);
seqTsFileResource.setUpgradeTsFileResourceCallBack(this::upgradeTsFileResourceCallBack);
seqTsFileResource.doUpgrade();
}
for (TsFileResource unseqTsFileResource : upgradeUnseqFileList) {
- unseqTsFileResource.setSeq(false);
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41738214)
Coverage decreased (-0.02%) to 68.193% when pulling **ea1a8ce0e57b26fa2a8e4672ed47c903a73d5388 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681045621
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/CompactionUtils.java
##########
@@ -67,6 +75,7 @@
public class CompactionUtils {
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42203085)
Coverage increased (+0.06%) to 67.289% when pulling **038bdf0270f2a1c62f0b1057c9c1957e66927f71 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43307413)
Coverage decreased (-0.005%) to 67.776% when pulling **cab2389b0f9039e798fa7263b51cbf0256cd0384 on THUMarkLau:new_compaction** into **cbbdc6caf51660e5817a4e9c854831d820315b72 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40844515)
Coverage increased (+0.1%) to 68.041% when pulling **6274179a213073e4fd5f364b08ad178f1f655026 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42272640)
Coverage increased (+0.04%) to 67.27% when pulling **e1faac2b9c1266ca6979c0082e302b5be6b4c92c on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41851302)
Coverage increased (+0.09%) to 68.293% when pulling **fbde36d1ee777bb8a38cf2344daddcd3bde7faf2 on THUMarkLau:new_compaction** into **1d0b6ff0893a8ea1e43ad66d1b589770bc898933 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43399434)
Coverage increased (+0.02%) to 67.815% when pulling **abee827b7d4bbe94532d46e416ce603479bf574b on THUMarkLau:new_compaction** into **5ab9e4abeb117201620588874e77a24b2bf98671 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r691842572
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1229,33 +1189,27 @@ private TsFileProcessor getOrCreateTsFileProcessorIntern(
// build new processor
res = newTsFileProcessor(sequence, timeRangeId);
tsFileProcessorTreeMap.put(timeRangeId, res);
- tsFileManagement.add(res.getTsFileResource(), sequence);
+ tsFileResourceManager.add(res.getTsFileResource(), sequence);
}
return res;
}
private TsFileProcessor newTsFileProcessor(boolean sequence, long timePartitionId)
throws IOException, DiskSpaceInsufficientException {
- DirectoryManager directoryManager = DirectoryManager.getInstance();
Review comment:
We abstracted out a TsFileNameGenerator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40843858)
Coverage increased (+0.2%) to 68.067% when pulling **6274179a213073e4fd5f364b08ad178f1f655026 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42229162)
Coverage increased (+0.03%) to 67.261% when pulling **c7b01da3f989a9fc4b6d9e42f31bf4da36d83e3b on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42224854)
Coverage increased (+0.04%) to 67.269% when pulling **26a6589e1adb5ed8311f986fac30e25213359a2d on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41134303)
Coverage increased (+0.7%) to 67.887% when pulling **d690367e5d800c6adaa2ad5b9729706af1c664f9 on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42208662)
Coverage increased (+0.04%) to 67.273% when pulling **2d76f3e0da966def582f7c63e849469e2ef1b17b on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43410168)
Coverage decreased (-0.06%) to 67.743% when pulling **e939bcc491361b668b42863f5a95dd422ffee27e on THUMarkLau:new_compaction** into **5ab9e4abeb117201620588874e77a24b2bf98671 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43306691)
Coverage increased (+0.006%) to 67.787% when pulling **cab2389b0f9039e798fa7263b51cbf0256cd0384 on THUMarkLau:new_compaction** into **cbbdc6caf51660e5817a4e9c854831d820315b72 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43722850)
Coverage increased (+0.005%) to 67.258% when pulling **57d46471674503b2b91d6fe9a6cdec7c39682bc8 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41742615)
Coverage increased (+0.001%) to 68.214% when pulling **4c5ce6fa0f390bcc514d372d55d0291389f3d648 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41423257)
Coverage increased (+1.0%) to 68.12% when pulling **450d6979b7c7a8e8889853186e40f92100db1abc on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42197551)
Coverage increased (+0.04%) to 67.267% when pulling **b4785e34d554573544a0cee5663daf6001e57aeb on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41561241)
Coverage increased (+1.0002%) to 68.148% when pulling **2bba9de7300e5397ea9be3eedd56d107301b6afe on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40925169)
Coverage decreased (-0.2%) to 67.672% when pulling **08741f6b7bd01b24acb926ed44476f0cde94893d on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43687793)
Coverage decreased (-0.01%) to 67.243% when pulling **41350dadc4fdab08bac829142d7314ab22eb325d on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] wangchao316 commented on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
wangchao316 commented on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-900215721
Good Job.
This is a big integration, and I suggest that we should be fully tested in one branch before incorporating it into the master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41829942)
Coverage increased (+0.08%) to 68.279% when pulling **76c46f3bb825f0ef1f900f1befc9947bde1bc588 on THUMarkLau:new_compaction** into **1d0b6ff0893a8ea1e43ad66d1b589770bc898933 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42500079)
Coverage increased (+0.08%) to 67.365% when pulling **43d8590eea13e0ae0a734eb79d04076f97da7fb1 on THUMarkLau:new_compaction** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41635072)
Coverage increased (+1.0%) to 68.108% when pulling **c432d68f699b35eb01bd9b926904d0a63847a42d on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40836853)
Coverage remained the same at 67.901% when pulling **a096b2ca62283964a7de0a2062622c814b57bdd9 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41704762)
Coverage decreased (-0.02%) to 68.193% when pulling **31958aa7c6c05a59e99de7f6fc2f9673b94f10c7 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41750126)
Coverage increased (+0.09%) to 68.286% when pulling **eca71a8edf4fd50b90faffac9a92c2b060e73a2a on THUMarkLau:new_compaction** into **1d0b6ff0893a8ea1e43ad66d1b589770bc898933 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43399245)
Coverage decreased (-0.01%) to 67.788% when pulling **abee827b7d4bbe94532d46e416ce603479bf574b on THUMarkLau:new_compaction** into **5ab9e4abeb117201620588874e77a24b2bf98671 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43473338)
Coverage decreased (-0.01%) to 67.731% when pulling **393e0c41334eabf086831470bf477c96dab68612 on THUMarkLau:new_compaction** into **c662a3e86de46aecc56236f0c2b693a2c479f38d on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43473372)
Coverage increased (+0.02%) to 67.761% when pulling **393e0c41334eabf086831470bf477c96dab68612 on THUMarkLau:new_compaction** into **c662a3e86de46aecc56236f0c2b693a2c479f38d on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681043251
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionTaskManager.java
##########
@@ -159,23 +137,32 @@ public ServiceType getID() {
return ServiceType.COMPACTION_SERVICE;
}
- public void submitTask(String storageGroupName, Callable<Void> compactionMergeTask)
+ public void submitTask(
+ String fullStorageGroupName, long timePartition, Callable<Void> compactionMergeTask)
throws RejectedExecutionException {
if (pool != null && !pool.isTerminated()) {
- Future<Void> future = pool.submit(compactionMergeTask);
- storageGroupTasks
- .computeIfAbsent(storageGroupName, k -> new ConcurrentSkipListSet<>())
- .add(future);
+ synchronized (CompactionScheduler.currentTaskNum) {
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41829563)
Coverage increased (+0.1%) to 68.304% when pulling **76c46f3bb825f0ef1f900f1befc9947bde1bc588 on THUMarkLau:new_compaction** into **1d0b6ff0893a8ea1e43ad66d1b589770bc898933 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40925085)
Coverage decreased (-0.2%) to 67.684% when pulling **08741f6b7bd01b24acb926ed44476f0cde94893d on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41114912)
Coverage increased (+0.8%) to 67.927% when pulling **b20396a56e530851b03695c9d531d54a97741b69 on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43673446)
Coverage decreased (-0.006%) to 67.248% when pulling **126c7187909229790393d394b2af9409961f1fd0 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43635066)
Coverage decreased (-0.2%) to 67.229% when pulling **5084039ff02a2d5409d5c6033ec507e3f699a7f5 on THUMarkLau:new_compaction** into **e4b7f64deb54b3fc186424cf969a68bff23a6fc7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42236566)
Coverage increased (+0.04%) to 67.266% when pulling **9c15c4d691c0cb96f519e438b100847839c7a403 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42203498)
Coverage increased (+0.06%) to 67.292% when pulling **038bdf0270f2a1c62f0b1057c9c1957e66927f71 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40955485)
Coverage decreased (-0.005%) to 67.143% when pulling **142fe7ad4148389396aba2841d4ab67d474a79fa on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42847934)
Coverage decreased (-0.05%) to 67.375% when pulling **012c3b169febb1c9ea070e673c415152c3a3b3fd on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42777856)
Coverage decreased (-0.02%) to 67.403% when pulling **b680eb5a91f0388382e0417141e58eb2a4173e2d on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42822148)
Coverage decreased (-0.03%) to 67.39% when pulling **646a35093060cb67bb57183e7df2446434d50ca8 on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43639377)
Coverage decreased (-0.03%) to 67.222% when pulling **04389e8f930f7079a122923cf2f9038e5f8cf188 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43699862)
Coverage decreased (-0.02%) to 67.236% when pulling **95bab5227c1ba23dff7a2a955a748b44c7da9536 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
qiaojialin merged pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41028790)
Coverage increased (+0.2%) to 67.355% when pulling **2a12778ded36ab0acb685c969ff675e62b58a30e on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41856565)
Coverage decreased (-0.005%) to 68.281% when pulling **d2dc97ac57e16179a368b7c4ba84e4f4facb9f2a on THUMarkLau:new_compaction** into **e3171bd7e0203923da0cfad3b9d8ebddd8211d70 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681048178
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResourceManager.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.exception.WriteLockFailedException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileResourceManager {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TsFileResourceManager.class);
+ private String storageGroupName;
+ private String virtualStorageGroup;
+ private String storageGroupDir;
+
+ /** Serialize queries, delete resource files, compaction cleanup files */
+ private final ReadWriteLock resourceListLock = new ReentrantReadWriteLock();
+
+ private String writeLockHolder;
+ // time partition -> double linked list of tsfiles
+ private Map<Long, TsFileResourceList> sequenceFiles = new TreeMap<>();
+ private Map<Long, TsFileResourceList> unsequenceFiles = new TreeMap<>();
+
+ private List<TsFileResource> sequenceRecoverTsFileResources = new ArrayList<>();
+ private List<TsFileResource> unsequenceRecoverTsFileResources = new ArrayList<>();
+
+ public TsFileResourceManager(
+ String storageGroupName, String virtualStorageGroup, String storageGroupDir) {
+ this.storageGroupName = storageGroupName;
+ this.storageGroupDir = storageGroupDir;
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getTsFileList(boolean sequence) {
+ readLock();
+ try {
+ List<TsFileResource> allResources = new ArrayList<>();
+ Map<Long, TsFileResourceList> chosenMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : chosenMap.entrySet()) {
+ allResources.addAll(entry.getValue().getArrayList());
+ }
+ return allResources;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public TsFileResourceList getSequenceListByTimePartition(long timePartition) {
+ return sequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public TsFileResourceList getUnsequenceListByTimePartition(long timePartition) {
+ return unsequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public Iterator<TsFileResource> getIterator(boolean sequence) {
+ readLock();
+ try {
+ return getTsFileList(sequence).iterator();
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void remove(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("remove");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (entry.getValue().contains(tsFileResource)) {
+ entry.getValue().remove(tsFileResource);
+ break;
+ }
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void removeAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ for (TsFileResource resource : tsFileResourceList) {
+ remove(resource, sequence);
+ }
+ }
+
+ /**
+ * insert tsFileResource to a target pos(targetPos = insertPos + 1) e.g. if insertPos = 0, then to
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -393,18 +394,17 @@ public StorageGroupProcessor(
this.fileFlushPolicy = fileFlushPolicy;
storageGroupSysDir = SystemFileFactory.INSTANCE.getFile(systemDir, virtualStorageGroupId);
+ this.tsFileResourceManager =
+ new TsFileResourceManager(
+ logicalStorageGroupName, virtualStorageGroupId, storageGroupSysDir.getPath());
if (storageGroupSysDir.mkdirs()) {
logger.info(
"Storage Group system Directory {} doesn't exist, create it",
storageGroupSysDir.getPath());
} else if (!storageGroupSysDir.exists()) {
logger.error("create Storage Group system Directory {} failed", storageGroupSysDir.getPath());
}
- this.tsFileManagement =
- IoTDBDescriptor.getInstance()
- .getConfig()
- .getCompactionStrategy()
- .getTsFileManagement(logicalStorageGroupName, storageGroupSysDir.getAbsolutePath());
+ // TODO: new TsFileManagement
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1307,8 +1261,8 @@ private String getNewTsFileName(long timePartitionId) {
return getNewTsFileName(System.currentTimeMillis(), version, 0, 0);
}
- private String getNewTsFileName(long time, long version, int mergeCnt, int unSeqMergeCnt) {
- return TsFileResource.getNewTsFileName(time, version, mergeCnt, unSeqMergeCnt);
+ private String getNewTsFileName(long time, long version, int mergeCnt, int unseqMergeCnt) {
Review comment:
fixed
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/compaction/utils/CompactionOverlapType.java
##########
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction.utils;
+
+public enum CompactionOverlapType {
+ FILE_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_OVERLAP_PAGE_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_OVERLAP_PAGE_OVERLAP
Review comment:
fixed
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/compaction/utils/CompactionOverlapType.java
##########
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction.utils;
+
+public enum CompactionOverlapType {
+ FILE_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_OVERLAP_PAGE_NO_OVERLAP,
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1137,6 +1139,7 @@ private void loadFile(File file, OperateFilePlan plan) throws QueryProcessExcept
StorageEngine.getInstance().loadNewTsFile(resource);
}
} catch (Exception e) {
+ e.printStackTrace();
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResourceManager.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.exception.WriteLockFailedException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileResourceManager {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TsFileResourceManager.class);
+ private String storageGroupName;
+ private String virtualStorageGroup;
+ private String storageGroupDir;
+
+ /** Serialize queries, delete resource files, compaction cleanup files */
+ private final ReadWriteLock resourceListLock = new ReentrantReadWriteLock();
+
+ private String writeLockHolder;
+ // time partition -> double linked list of tsfiles
+ private Map<Long, TsFileResourceList> sequenceFiles = new TreeMap<>();
+ private Map<Long, TsFileResourceList> unsequenceFiles = new TreeMap<>();
+
+ private List<TsFileResource> sequenceRecoverTsFileResources = new ArrayList<>();
+ private List<TsFileResource> unsequenceRecoverTsFileResources = new ArrayList<>();
+
+ public TsFileResourceManager(
+ String storageGroupName, String virtualStorageGroup, String storageGroupDir) {
+ this.storageGroupName = storageGroupName;
+ this.storageGroupDir = storageGroupDir;
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getTsFileList(boolean sequence) {
+ readLock();
+ try {
+ List<TsFileResource> allResources = new ArrayList<>();
+ Map<Long, TsFileResourceList> chosenMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : chosenMap.entrySet()) {
+ allResources.addAll(entry.getValue().getArrayList());
+ }
+ return allResources;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public TsFileResourceList getSequenceListByTimePartition(long timePartition) {
+ return sequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public TsFileResourceList getUnsequenceListByTimePartition(long timePartition) {
+ return unsequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public Iterator<TsFileResource> getIterator(boolean sequence) {
+ readLock();
+ try {
+ return getTsFileList(sequence).iterator();
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void remove(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("remove");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (entry.getValue().contains(tsFileResource)) {
+ entry.getValue().remove(tsFileResource);
+ break;
+ }
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void removeAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ for (TsFileResource resource : tsFileResourceList) {
+ remove(resource, sequence);
+ }
+ }
+
+ /**
+ * insert tsFileResource to a target pos(targetPos = insertPos + 1) e.g. if insertPos = 0, then to
+ * the first, if insert Pos = 1, then to the second.
+ */
+ public void insert(TsFileResource tsFileResource, boolean sequence, int insertPos) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ TsFileResourceList tsFileResources =
+ selectedMap.computeIfAbsent(
+ tsFileResource.getTimePartition(), o -> new TsFileResourceList());
+ tsFileResources.set(insertPos, tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void add(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ selectedMap
+ .computeIfAbsent(tsFileResource.getTimePartition(), o -> new TsFileResourceList())
+ .add(tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void addRecover(TsFileResource tsFileResource, boolean sequence) {
+ if (sequence) {
+ sequenceRecoverTsFileResources.add(tsFileResource);
+ } else {
+ unsequenceRecoverTsFileResources.add(tsFileResource);
+ }
+ }
+
+ public void addAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ writeLock("add");
+ try {
+ for (TsFileResource resource : tsFileResourceList) {
+ add(resource, sequence);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public boolean contains(TsFileResource tsFileResource, boolean sequence) {
+ readLock();
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ TsFileResourceList list = selectedMap.getOrDefault(tsFileResource.getTimePartition(), null);
+ return list != null && list.contains(tsFileResource);
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void clear() {
+ writeLock("clear");
+ try {
+ sequenceFiles.clear();
+ unsequenceFiles.clear();
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public boolean isEmpty(boolean sequence) {
+ readLock();
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (!entry.getValue().isEmpty()) {
+ return false;
+ }
+ }
+ return true;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public int size(boolean sequence) {
+ readLock();
+ try {
+ int totalSize = 0;
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ totalSize += entry.getValue().size();
+ }
+ return totalSize;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void readLock() {
+ resourceListLock.readLock().lock();
+ }
+
+ public void readUnlock() {
+ resourceListLock.readLock().unlock();
+ }
+
+ public void writeLock(String holder) {
+ resourceListLock.writeLock().lock();
+ writeLockHolder = holder;
+ }
+
+ /**
+ * Acquire write lock with timeout, {@link WriteLockFailedException} will be thrown after timeout.
+ * The unit of timeout is ms.
+ */
+ public void writeLockWithTimeout(String holder, long timeout) throws WriteLockFailedException {
+ try {
+ if (resourceListLock.writeLock().tryLock(timeout, TimeUnit.MILLISECONDS)) {
+ writeLockHolder = holder;
+ } else {
+ throw new WriteLockFailedException(
+ String.format("cannot get write lock in %d ms", timeout));
+ }
+ } catch (InterruptedException e) {
+ LOGGER.warn(e.getMessage(), e);
+ Thread.interrupted();
+ throw new WriteLockFailedException("thread is interrupted");
+ }
+ }
+
+ public void writeUnlock() {
+ resourceListLock.writeLock().unlock();
+ writeLockHolder = "";
+ }
+
+ public String getStorageGroupName() {
+ return storageGroupName;
+ }
+
+ public String getStorageGroupDir() {
+ return storageGroupDir;
+ }
+
+ public void setStorageGroupDir(String storageGroupDir) {
+ this.storageGroupDir = storageGroupDir;
+ }
+
+ public Set<Long> getTimePartitions() {
+ readLock();
+ try {
+ Set<Long> timePartitions = new HashSet<>(sequenceFiles.keySet());
+ return timePartitions;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public String getVirtualStorageGroup() {
+ return virtualStorageGroup;
+ }
+
+ public void setVirtualStorageGroup(String virtualStorageGroup) {
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getSequenceRecoverTsFileResources() {
+ return sequenceRecoverTsFileResources;
+ }
+
+ public List<TsFileResource> getUnsequenceRecoverTsFileResources() {
+ return unsequenceRecoverTsFileResources;
+ }
+
+ // ({systemTime}-{versionNum}-{compactionNum}-{mergeNum}.tsfile)
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
HTHou commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r657593163
##########
File path: example/tsfile/src/main/java/org/apache/iotdb/tsfile/TsFileSequenceRead.java
##########
@@ -46,7 +46,7 @@
"squid:S106"
}) // Suppress high Cognitive Complexity and Standard outputs warning
public static void main(String[] args) throws IOException {
- String filename = "test.tsfile";
+ String filename = "C:\\Users\\MARKLAU\\Desktop\\0-0-1-0.tsfile";
Review comment:
```suggestion
String filename = "test.tsfile";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40875274)
Coverage increased (+0.2%) to 68.076% when pulling **c3eb8a4cf60ea8945bde18bbb6e1eeb8ecf99d22 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40955847)
Coverage decreased (-0.005%) to 67.143% when pulling **142fe7ad4148389396aba2841d4ab67d474a79fa on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41742802)
Coverage decreased (-0.03%) to 68.188% when pulling **4c5ce6fa0f390bcc514d372d55d0291389f3d648 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42243748)
Coverage increased (+0.04%) to 67.267% when pulling **56415dfb9cbe1340fa3c462344f3760184377038 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43674972)
Coverage decreased (-0.006%) to 67.248% when pulling **3d877e5252bead0d5f02f1d72656c82411b90684 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43693584)
Coverage increased (+0.002%) to 67.255% when pulling **d6378b4b349c2129448c57ff4fb8d8ad35b89d60 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43639631)
Coverage decreased (-0.006%) to 67.247% when pulling **04389e8f930f7079a122923cf2f9038e5f8cf188 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43722746)
Coverage increased (+0.01%) to 67.267% when pulling **fc3ef51f493c61ac8f342f42d7c5ed1892f61e3b on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42208172)
Coverage increased (+0.03%) to 67.256% when pulling **2d76f3e0da966def582f7c63e849469e2ef1b17b on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42192226)
Coverage increased (+0.05%) to 67.278% when pulling **895e246cf7ec50ec62dfcc32b14a9ee2ff82bb4f on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40955612)
Coverage increased (+0.01%) to 67.16% when pulling **142fe7ad4148389396aba2841d4ab67d474a79fa on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43405848)
Coverage decreased (-0.04%) to 67.755% when pulling **7883f87ac64543647548eb19e5b7ddb054ad827d on THUMarkLau:new_compaction** into **5ab9e4abeb117201620588874e77a24b2bf98671 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42202318)
Coverage increased (+0.05%) to 67.281% when pulling **389b6f4b58897f339850e0451f1010d1c5ad5d7b on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41711140)
Coverage increased (+0.02%) to 68.235% when pulling **c51ce1c87e9c8a0541dc4ed06ee010fa97bf5a41 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40837436)
Coverage decreased (-0.02%) to 67.877% when pulling **3ca922fd9aade2ad3e4fa1beedf40f3372299076 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41711079)
Coverage increased (+0.004%) to 68.217% when pulling **c51ce1c87e9c8a0541dc4ed06ee010fa97bf5a41 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41426443)
Coverage increased (+1.005%) to 68.153% when pulling **6a4f5b49b9186f4289d2694b8b93b6069d4334b7 on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681042935
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/logtypes/CloseFileLog.java
##########
@@ -90,7 +90,7 @@ public long getPartitionId() {
@Override
public String toString() {
return "CloseFileLog{"
- + "storageGroupName='"
+ + "fullStorageGroupName='"
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionScheduler.java
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.compaction.cross.AbstractCrossSpaceCompactionSelector;
+import org.apache.iotdb.db.engine.compaction.cross.CrossSpaceCompactionTaskFactory;
+import org.apache.iotdb.db.engine.compaction.inner.AbstractInnerSpaceCompactionSelector;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTaskFactory;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionSelector;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceManager;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * CompactionScheduler schedule and submit the compaction task periodically, and it count the total
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42780135)
Coverage decreased (-0.009%) to 67.412% when pulling **5127a32c5d0b5f8b1cf7f94a1363cc315c98f4bd on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r678124825
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/cross/CrossCompactionStrategy.java
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.compaction.cross;
+
+import org.apache.iotdb.db.engine.compaction.cross.inplace.InplaceCompactionRecoverTask;
+import org.apache.iotdb.db.engine.compaction.cross.inplace.InplaceCompactionSelector;
+import org.apache.iotdb.db.engine.compaction.cross.inplace.InplaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.cross.inplace.manage.CrossSpaceMergeResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+
+import java.io.File;
+import java.util.List;
+
+public enum CrossCompactionStrategy {
+ INPLACE_COMPACTION;
+
+ public AbstractCrossSpaceCompactionTask getCompactionTask(
+ String logicalStorageGroupName,
+ String virtualStorageGroupName,
+ long timePartitionId,
+ CrossSpaceMergeResource mergeResource,
+ String storageGroupDir,
+ TsFileResourceList seqTsFileResourceList,
+ TsFileResourceList unSeqTsFileResourceList,
+ List<TsFileResource> selectedSeqTsFileResourceList,
+ List<TsFileResource> selectedUnSeqTsFileResourceList,
+ int concurrentMergeCount) {
+ switch (this) {
+ case INPLACE_COMPACTION:
+ default:
+ return new InplaceCompactionTask(
+ logicalStorageGroupName,
+ virtualStorageGroupName,
+ timePartitionId,
+ mergeResource,
+ storageGroupDir,
+ seqTsFileResourceList,
+ unSeqTsFileResourceList,
+ selectedSeqTsFileResourceList,
+ selectedUnSeqTsFileResourceList,
+ concurrentMergeCount);
+ }
+ }
+
+ public AbstractCrossSpaceCompactionTask getCompactionRecoverTask(
Review comment:
```suggestion
public AbstractCrossSpaceCompactionRecoverTask getCompactionRecoverTask(
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1829,15 +1771,23 @@ public void delete(PartialPath path, long startTime, long endTime, long planInde
logDeletion(startTime, endTime, path);
Deletion deletion = new Deletion(path, MERGE_MOD_START_VERSION_NUM, startTime, endTime);
- if (tsFileManagement.mergingModification != null) {
- tsFileManagement.mergingModification.write(deletion);
- updatedModFiles.add(tsFileManagement.mergingModification);
- }
+ // if (tsFileResourceManager.mergingModification != null) {
Review comment:
remove
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
##########
@@ -438,7 +439,14 @@ public boolean resourceFileExists() {
public synchronized ModificationFile getModFile() {
Review comment:
remove synchronized and use double check
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionTaskManager.java
##########
@@ -159,23 +137,32 @@ public ServiceType getID() {
return ServiceType.COMPACTION_SERVICE;
}
- public void submitTask(String storageGroupName, Callable<Void> compactionMergeTask)
+ public void submitTask(
+ String fullStorageGroupName, long timePartition, Callable<Void> compactionMergeTask)
throws RejectedExecutionException {
if (pool != null && !pool.isTerminated()) {
- Future<Void> future = pool.submit(compactionMergeTask);
- storageGroupTasks
- .computeIfAbsent(storageGroupName, k -> new ConcurrentSkipListSet<>())
- .add(future);
+ synchronized (CompactionScheduler.currentTaskNum) {
Review comment:
synchronize this mehod?
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/task/AbstractCompactionTask.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction.task;
+
+import org.apache.iotdb.db.engine.compaction.CompactionScheduler;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.Callable;
+
+import static org.apache.iotdb.db.engine.compaction.CompactionScheduler.currentTaskNum;
+
+/**
+ * AbstractCompactionTask is the base class for all compaction task, it carries out the execution of
+ * compaction. AbstractCompactionTask uses a template method, it execute the abstract function
+ * <i>doCompaction</i> implemented by subclass, and decrease the currentTaskNum in
+ * CompactionScheduler when the <i>doCompaction</i> finish.
+ */
+public abstract class AbstractCompactionTask implements Callable<Void> {
+ private static final Logger LOGGER = LoggerFactory.getLogger(AbstractCompactionTask.class);
+ protected String fullStorageGroupName;
+ protected long timePartition;
+
+ public AbstractCompactionTask(String fullStorageGroupName, long timePartition) {
+ this.fullStorageGroupName = fullStorageGroupName;
+ this.timePartition = timePartition;
+ }
+
+ protected abstract void doCompaction() throws Exception;
+
+ @Override
+ public Void call() throws Exception {
+ try {
+ doCompaction();
+ } catch (Exception e) {
+ LOGGER.warn(e.getMessage(), e);
Review comment:
add some information
meet error in doCompaction
```suggestion
LOGGER.error(e.getMessage(), e);
```
##########
File path: cluster/src/main/java/org/apache/iotdb/cluster/log/logtypes/CloseFileLog.java
##########
@@ -90,7 +90,7 @@ public long getPartitionId() {
@Override
public String toString() {
return "CloseFileLog{"
- + "storageGroupName='"
+ + "fullStorageGroupName='"
Review comment:
is this meant?
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -393,18 +394,17 @@ public StorageGroupProcessor(
this.fileFlushPolicy = fileFlushPolicy;
storageGroupSysDir = SystemFileFactory.INSTANCE.getFile(systemDir, virtualStorageGroupId);
+ this.tsFileResourceManager =
+ new TsFileResourceManager(
+ logicalStorageGroupName, virtualStorageGroupId, storageGroupSysDir.getPath());
if (storageGroupSysDir.mkdirs()) {
logger.info(
"Storage Group system Directory {} doesn't exist, create it",
storageGroupSysDir.getPath());
} else if (!storageGroupSysDir.exists()) {
logger.error("create Storage Group system Directory {} failed", storageGroupSysDir.getPath());
}
- this.tsFileManagement =
- IoTDBDescriptor.getInstance()
- .getConfig()
- .getCompactionStrategy()
- .getTsFileManagement(logicalStorageGroupName, storageGroupSysDir.getAbsolutePath());
+ // TODO: new TsFileManagement
Review comment:
remove
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionScheduler.java
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.compaction.cross.AbstractCrossSpaceCompactionSelector;
+import org.apache.iotdb.db.engine.compaction.cross.CrossSpaceCompactionTaskFactory;
+import org.apache.iotdb.db.engine.compaction.inner.AbstractInnerSpaceCompactionSelector;
+import org.apache.iotdb.db.engine.compaction.inner.InnerSpaceCompactionTaskFactory;
+import org.apache.iotdb.db.engine.compaction.task.AbstractCompactionSelector;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceManager;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * CompactionScheduler schedule and submit the compaction task periodically, and it count the total
Review comment:
```suggestion
* CompactionScheduler schedules and submits the compaction task periodically, and it counts the total
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/CompactionUtils.java
##########
@@ -477,9 +484,79 @@ private static void modifyChunkMetaDataWithCache(
for (Modification modification : modifications) {
if (modification.getPath().matchFullPath(seriesPath)) {
seriesModifications.add(modification);
- usedModifications.add(modification);
}
}
modifyChunkMetaData(chunkMetadataList, seriesModifications);
}
+
+ private static boolean isFileListHasModifications(
+ Set<TsFileSequenceReader> readers, Map<String, List<Modification>> modificationCache) {
+ for (TsFileSequenceReader reader : readers) {
+ if (!getModifications(reader, modificationCache).isEmpty()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static List<Modification> getModifications(
+ TsFileSequenceReader reader, Map<String, List<Modification>> modificationCache) {
+ return modificationCache.computeIfAbsent(
+ reader.getFileName(),
+ fileName ->
+ new LinkedList<>(
+ new ModificationFile(fileName + ModificationFile.FILE_SUFFIX).getModifications()));
+ }
+
+ public static ModificationFile getModificationByPath(String filePath) {
Review comment:
remove
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionTaskManager.java
##########
@@ -94,25 +89,8 @@ public void waitAndStop(long milliseconds) {
@TestOnly
public void waitAllCompactionFinish() {
if (pool != null) {
- File sgDir =
- FSFactoryProducer.getFSFactory()
- .getFile(
- FilePathUtils.regularizePath(
- IoTDBDescriptor.getInstance().getConfig().getSystemDir())
- + "storage_groups");
- File[] subDirList = sgDir.listFiles();
- if (subDirList != null) {
- for (File subDir : subDirList) {
- while (FSFactoryProducer.getFSFactory()
- .getFile(
- subDir.getAbsoluteFile()
- + File.separator
- + subDir.getName()
- + COMPACTION_LOG_NAME)
- .exists()) {
- // wait
- }
- }
+ while (CompactionScheduler.currentTaskNum.get() > 0) {
+ // wait
Review comment:
sleep
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1229,33 +1189,27 @@ private TsFileProcessor getOrCreateTsFileProcessorIntern(
// build new processor
res = newTsFileProcessor(sequence, timeRangeId);
tsFileProcessorTreeMap.put(timeRangeId, res);
- tsFileManagement.add(res.getTsFileResource(), sequence);
+ tsFileResourceManager.add(res.getTsFileResource(), sequence);
}
return res;
}
private TsFileProcessor newTsFileProcessor(boolean sequence, long timePartitionId)
throws IOException, DiskSpaceInsufficientException {
- DirectoryManager directoryManager = DirectoryManager.getInstance();
Review comment:
compare with master
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/CompactionUtils.java
##########
@@ -67,6 +75,7 @@
public class CompactionUtils {
Review comment:
```suggestion
public class InnerSpaceCompactionUtils {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1307,8 +1261,8 @@ private String getNewTsFileName(long timePartitionId) {
return getNewTsFileName(System.currentTimeMillis(), version, 0, 0);
}
- private String getNewTsFileName(long time, long version, int mergeCnt, int unSeqMergeCnt) {
- return TsFileResource.getNewTsFileName(time, version, mergeCnt, unSeqMergeCnt);
+ private String getNewTsFileName(long time, long version, int mergeCnt, int unseqMergeCnt) {
Review comment:
```suggestion
private String getNewTsFileName(long time, long version, int mergeCnt, int unseqCompactionCnt) {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2124,12 +2050,10 @@ public int countUpgradeFiles() {
/** upgrade all files belongs to this storage group */
public void upgrade() {
for (TsFileResource seqTsFileResource : upgradeSeqFileList) {
- seqTsFileResource.setSeq(true);
Review comment:
recover this
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/CompactionUtils.java
##########
@@ -477,9 +484,79 @@ private static void modifyChunkMetaDataWithCache(
for (Modification modification : modifications) {
if (modification.getPath().matchFullPath(seriesPath)) {
seriesModifications.add(modification);
- usedModifications.add(modification);
}
}
modifyChunkMetaData(chunkMetadataList, seriesModifications);
}
+
+ private static boolean isFileListHasModifications(
Review comment:
check modification in chunk level
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1449,9 +1403,9 @@ public void syncDeleteDataFiles() {
syncCloseAllWorkingTsFileProcessors();
// normally, mergingModification is just need to be closed by after a merge task is finished.
// we close it here just for IT test.
- if (this.tsFileManagement.mergingModification != null) {
- this.tsFileManagement.mergingModification.close();
- }
+ // if (this.tsFileResourceManager.mergingModification != null) {
Review comment:
remove
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1902,11 +1852,21 @@ private void deleteDataInFiles(
continue;
}
- deletion.setFileOffset(tsFileResource.getTsFileSize());
- // write deletion into modification file
- tsFileResource.getModFile().write(deletion);
- // remember to close mod file
- tsFileResource.getModFile().close();
+ if (tsFileResource.isMerging) {
Review comment:
add design doc for delete in compaction
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
Review comment:
```suggestion
public static String generateNewTsFilePathWithMkdir(
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -2124,12 +2050,10 @@ public int countUpgradeFiles() {
/** upgrade all files belongs to this storage group */
public void upgrade() {
for (TsFileResource seqTsFileResource : upgradeSeqFileList) {
- seqTsFileResource.setSeq(true);
seqTsFileResource.setUpgradeTsFileResourceCallBack(this::upgradeTsFileResourceCallBack);
seqTsFileResource.doUpgrade();
}
for (TsFileResource unseqTsFileResource : upgradeUnseqFileList) {
- unseqTsFileResource.setSeq(false);
Review comment:
recover
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
Review comment:
```suggestion
throws DiskSpaceInsufficientException {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
Review comment:
```suggestion
public static TsFileResource increaseCrossCompactionCnt(TsFileResource tsFileResource)
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
##########
@@ -813,6 +813,11 @@ public void updatePlanIndexes(long planIndex) {
}
}
+ public static int getCompactionCount(String fileName) throws IOException {
Review comment:
```suggestion
public static int getInnerCompactionCount(String fileName) throws IOException {
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
##########
@@ -438,7 +439,14 @@ public boolean resourceFileExists() {
public synchronized ModificationFile getModFile() {
if (modFile == null) {
- modFile = new ModificationFile(file.getPath() + ModificationFile.FILE_SUFFIX);
+ modFile = ModificationFile.getNormalMods(this);
+ }
+ return modFile;
+ }
+
+ public synchronized ModificationFile getCompactionModFile() {
Review comment:
remove synchronized and use double check
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
+ throws IOException {
+ File tsFile = tsFileResource.getTsFile();
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFileResource.getTsFile().getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ tsFileResource.setFile(
+ new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX));
+ return tsFileResource;
+ }
+
+ public static File modifyTsFileNameUnseqMergCnt(File tsFile) throws IOException {
Review comment:
remove
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
+ throws IOException {
+ File tsFile = tsFileResource.getTsFile();
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFileResource.getTsFile().getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ tsFileResource.setFile(
+ new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX));
+ return tsFileResource;
+ }
+
+ public static File modifyTsFileNameUnseqMergCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File modifyTsFileNameMergeCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setMergeCnt(tsFileName.getMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File getInnerCompactionFileName(List<TsFileResource> tsFileResources)
+ throws IOException {
+ long minTime = Long.MAX_VALUE;
+ long minVersion = Long.MAX_VALUE;
+ long maxInnerMergeCount = Long.MIN_VALUE;
+ long maxCrossMergeCount = Long.MIN_VALUE;
+ for (TsFileResource resource : tsFileResources) {
+ TsFileName tsFileName = getTsFileName(resource.getTsFile().getName());
+ minTime = Math.min(tsFileName.time, minTime);
+ minVersion = Math.min(tsFileName.version, minVersion);
+ maxInnerMergeCount = Math.max(tsFileName.mergeCnt, maxInnerMergeCount);
+ maxCrossMergeCount = Math.max(tsFileName.unSeqMergeCnt, maxCrossMergeCount);
+ }
+ return new File(
+ tsFileResources.get(0).getTsFile().getParent(),
+ minTime
+ + FILE_NAME_SEPARATOR
+ + minVersion
+ + FILE_NAME_SEPARATOR
+ + (maxInnerMergeCount + 1)
+ + FILE_NAME_SEPARATOR
+ + maxCrossMergeCount
+ + TSFILE_SUFFIX);
+ }
+
+ public static class TsFileName {
+
+ private long time;
+ private long version;
+ private int mergeCnt;
+ private int unSeqMergeCnt;
Review comment:
```suggestion
private int crossCompactionCnt;
```
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/compaction/utils/CompactionOverlapType.java
##########
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction.utils;
+
+public enum CompactionOverlapType {
+ FILE_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_OVERLAP_PAGE_NO_OVERLAP,
Review comment:
```suggestion
CHUNK_OVERLAP_PAGE_NO_OVERLAP,
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResourceManager.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.exception.WriteLockFailedException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileResourceManager {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TsFileResourceManager.class);
+ private String storageGroupName;
+ private String virtualStorageGroup;
+ private String storageGroupDir;
+
+ /** Serialize queries, delete resource files, compaction cleanup files */
+ private final ReadWriteLock resourceListLock = new ReentrantReadWriteLock();
+
+ private String writeLockHolder;
+ // time partition -> double linked list of tsfiles
+ private Map<Long, TsFileResourceList> sequenceFiles = new TreeMap<>();
+ private Map<Long, TsFileResourceList> unsequenceFiles = new TreeMap<>();
+
+ private List<TsFileResource> sequenceRecoverTsFileResources = new ArrayList<>();
+ private List<TsFileResource> unsequenceRecoverTsFileResources = new ArrayList<>();
+
+ public TsFileResourceManager(
+ String storageGroupName, String virtualStorageGroup, String storageGroupDir) {
+ this.storageGroupName = storageGroupName;
+ this.storageGroupDir = storageGroupDir;
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getTsFileList(boolean sequence) {
+ readLock();
+ try {
+ List<TsFileResource> allResources = new ArrayList<>();
+ Map<Long, TsFileResourceList> chosenMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : chosenMap.entrySet()) {
+ allResources.addAll(entry.getValue().getArrayList());
+ }
+ return allResources;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public TsFileResourceList getSequenceListByTimePartition(long timePartition) {
+ return sequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public TsFileResourceList getUnsequenceListByTimePartition(long timePartition) {
+ return unsequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public Iterator<TsFileResource> getIterator(boolean sequence) {
+ readLock();
+ try {
+ return getTsFileList(sequence).iterator();
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void remove(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("remove");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (entry.getValue().contains(tsFileResource)) {
+ entry.getValue().remove(tsFileResource);
+ break;
+ }
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void removeAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ for (TsFileResource resource : tsFileResourceList) {
+ remove(resource, sequence);
+ }
+ }
+
+ /**
+ * insert tsFileResource to a target pos(targetPos = insertPos + 1) e.g. if insertPos = 0, then to
+ * the first, if insert Pos = 1, then to the second.
+ */
+ public void insert(TsFileResource tsFileResource, boolean sequence, int insertPos) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ TsFileResourceList tsFileResources =
+ selectedMap.computeIfAbsent(
+ tsFileResource.getTimePartition(), o -> new TsFileResourceList());
+ tsFileResources.set(insertPos, tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void add(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ selectedMap
+ .computeIfAbsent(tsFileResource.getTimePartition(), o -> new TsFileResourceList())
+ .add(tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void addRecover(TsFileResource tsFileResource, boolean sequence) {
+ if (sequence) {
+ sequenceRecoverTsFileResources.add(tsFileResource);
+ } else {
+ unsequenceRecoverTsFileResources.add(tsFileResource);
+ }
+ }
+
+ public void addAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ writeLock("add");
+ try {
+ for (TsFileResource resource : tsFileResourceList) {
+ add(resource, sequence);
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public boolean contains(TsFileResource tsFileResource, boolean sequence) {
+ readLock();
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ TsFileResourceList list = selectedMap.getOrDefault(tsFileResource.getTimePartition(), null);
+ return list != null && list.contains(tsFileResource);
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void clear() {
+ writeLock("clear");
+ try {
+ sequenceFiles.clear();
+ unsequenceFiles.clear();
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public boolean isEmpty(boolean sequence) {
+ readLock();
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (!entry.getValue().isEmpty()) {
+ return false;
+ }
+ }
+ return true;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public int size(boolean sequence) {
+ readLock();
+ try {
+ int totalSize = 0;
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ totalSize += entry.getValue().size();
+ }
+ return totalSize;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void readLock() {
+ resourceListLock.readLock().lock();
+ }
+
+ public void readUnlock() {
+ resourceListLock.readLock().unlock();
+ }
+
+ public void writeLock(String holder) {
+ resourceListLock.writeLock().lock();
+ writeLockHolder = holder;
+ }
+
+ /**
+ * Acquire write lock with timeout, {@link WriteLockFailedException} will be thrown after timeout.
+ * The unit of timeout is ms.
+ */
+ public void writeLockWithTimeout(String holder, long timeout) throws WriteLockFailedException {
+ try {
+ if (resourceListLock.writeLock().tryLock(timeout, TimeUnit.MILLISECONDS)) {
+ writeLockHolder = holder;
+ } else {
+ throw new WriteLockFailedException(
+ String.format("cannot get write lock in %d ms", timeout));
+ }
+ } catch (InterruptedException e) {
+ LOGGER.warn(e.getMessage(), e);
+ Thread.interrupted();
+ throw new WriteLockFailedException("thread is interrupted");
+ }
+ }
+
+ public void writeUnlock() {
+ resourceListLock.writeLock().unlock();
+ writeLockHolder = "";
+ }
+
+ public String getStorageGroupName() {
+ return storageGroupName;
+ }
+
+ public String getStorageGroupDir() {
+ return storageGroupDir;
+ }
+
+ public void setStorageGroupDir(String storageGroupDir) {
+ this.storageGroupDir = storageGroupDir;
+ }
+
+ public Set<Long> getTimePartitions() {
+ readLock();
+ try {
+ Set<Long> timePartitions = new HashSet<>(sequenceFiles.keySet());
+ return timePartitions;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public String getVirtualStorageGroup() {
+ return virtualStorageGroup;
+ }
+
+ public void setVirtualStorageGroup(String virtualStorageGroup) {
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getSequenceRecoverTsFileResources() {
+ return sequenceRecoverTsFileResources;
+ }
+
+ public List<TsFileResource> getUnsequenceRecoverTsFileResources() {
+ return unsequenceRecoverTsFileResources;
+ }
+
+ // ({systemTime}-{versionNum}-{compactionNum}-{mergeNum}.tsfile)
Review comment:
```suggestion
// ({systemTime}-{versionNum}-{innerCompactionNum}-{crossCompactionNum}.tsfile)
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResourceManager.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.exception.WriteLockFailedException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileResourceManager {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TsFileResourceManager.class);
+ private String storageGroupName;
+ private String virtualStorageGroup;
+ private String storageGroupDir;
+
+ /** Serialize queries, delete resource files, compaction cleanup files */
+ private final ReadWriteLock resourceListLock = new ReentrantReadWriteLock();
+
+ private String writeLockHolder;
+ // time partition -> double linked list of tsfiles
+ private Map<Long, TsFileResourceList> sequenceFiles = new TreeMap<>();
+ private Map<Long, TsFileResourceList> unsequenceFiles = new TreeMap<>();
+
+ private List<TsFileResource> sequenceRecoverTsFileResources = new ArrayList<>();
+ private List<TsFileResource> unsequenceRecoverTsFileResources = new ArrayList<>();
+
+ public TsFileResourceManager(
+ String storageGroupName, String virtualStorageGroup, String storageGroupDir) {
+ this.storageGroupName = storageGroupName;
+ this.storageGroupDir = storageGroupDir;
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getTsFileList(boolean sequence) {
+ readLock();
+ try {
+ List<TsFileResource> allResources = new ArrayList<>();
+ Map<Long, TsFileResourceList> chosenMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : chosenMap.entrySet()) {
+ allResources.addAll(entry.getValue().getArrayList());
+ }
+ return allResources;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public TsFileResourceList getSequenceListByTimePartition(long timePartition) {
+ return sequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public TsFileResourceList getUnsequenceListByTimePartition(long timePartition) {
+ return unsequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public Iterator<TsFileResource> getIterator(boolean sequence) {
+ readLock();
+ try {
+ return getTsFileList(sequence).iterator();
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void remove(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("remove");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (entry.getValue().contains(tsFileResource)) {
+ entry.getValue().remove(tsFileResource);
+ break;
+ }
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void removeAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ for (TsFileResource resource : tsFileResourceList) {
+ remove(resource, sequence);
+ }
+ }
+
+ /**
+ * insert tsFileResource to a target pos(targetPos = insertPos + 1) e.g. if insertPos = 0, then to
Review comment:
```suggestion
* insert tsFileResource to a target pos(targetPos = insertPos) e.g. if insertPos = 0, then to
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
+ throws IOException {
+ File tsFile = tsFileResource.getTsFile();
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFileResource.getTsFile().getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ tsFileResource.setFile(
+ new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX));
+ return tsFileResource;
+ }
+
+ public static File modifyTsFileNameUnseqMergCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File modifyTsFileNameMergeCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setMergeCnt(tsFileName.getMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File getInnerCompactionFileName(List<TsFileResource> tsFileResources)
+ throws IOException {
+ long minTime = Long.MAX_VALUE;
+ long minVersion = Long.MAX_VALUE;
+ long maxInnerMergeCount = Long.MIN_VALUE;
+ long maxCrossMergeCount = Long.MIN_VALUE;
+ for (TsFileResource resource : tsFileResources) {
+ TsFileName tsFileName = getTsFileName(resource.getTsFile().getName());
+ minTime = Math.min(tsFileName.time, minTime);
+ minVersion = Math.min(tsFileName.version, minVersion);
+ maxInnerMergeCount = Math.max(tsFileName.mergeCnt, maxInnerMergeCount);
+ maxCrossMergeCount = Math.max(tsFileName.unSeqMergeCnt, maxCrossMergeCount);
+ }
+ return new File(
+ tsFileResources.get(0).getTsFile().getParent(),
+ minTime
+ + FILE_NAME_SEPARATOR
+ + minVersion
+ + FILE_NAME_SEPARATOR
+ + (maxInnerMergeCount + 1)
+ + FILE_NAME_SEPARATOR
+ + maxCrossMergeCount
+ + TSFILE_SUFFIX);
+ }
+
+ public static class TsFileName {
+
+ private long time;
+ private long version;
+ private int mergeCnt;
Review comment:
```suggestion
private int innerCompactionCnt;
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
Review comment:
check who use this previous
##########
File path: server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
##########
@@ -1137,6 +1139,7 @@ private void loadFile(File file, OperateFilePlan plan) throws QueryProcessExcept
StorageEngine.getInstance().loadNewTsFile(resource);
}
} catch (Exception e) {
+ e.printStackTrace();
Review comment:
use logger
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/compaction/utils/CompactionOverlapType.java
##########
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction.utils;
+
+public enum CompactionOverlapType {
+ FILE_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_OVERLAP_PAGE_NO_OVERLAP,
+ FILE_OVERLAP_CHUNK_OVERLAP_PAGE_OVERLAP
Review comment:
```suggestion
PAGE_OVERLAP
```
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResourceManager.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.exception.WriteLockFailedException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileResourceManager {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TsFileResourceManager.class);
+ private String storageGroupName;
+ private String virtualStorageGroup;
+ private String storageGroupDir;
+
+ /** Serialize queries, delete resource files, compaction cleanup files */
+ private final ReadWriteLock resourceListLock = new ReentrantReadWriteLock();
+
+ private String writeLockHolder;
+ // time partition -> double linked list of tsfiles
+ private Map<Long, TsFileResourceList> sequenceFiles = new TreeMap<>();
+ private Map<Long, TsFileResourceList> unsequenceFiles = new TreeMap<>();
+
+ private List<TsFileResource> sequenceRecoverTsFileResources = new ArrayList<>();
+ private List<TsFileResource> unsequenceRecoverTsFileResources = new ArrayList<>();
+
+ public TsFileResourceManager(
+ String storageGroupName, String virtualStorageGroup, String storageGroupDir) {
+ this.storageGroupName = storageGroupName;
+ this.storageGroupDir = storageGroupDir;
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getTsFileList(boolean sequence) {
+ readLock();
+ try {
+ List<TsFileResource> allResources = new ArrayList<>();
+ Map<Long, TsFileResourceList> chosenMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : chosenMap.entrySet()) {
+ allResources.addAll(entry.getValue().getArrayList());
+ }
+ return allResources;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public TsFileResourceList getSequenceListByTimePartition(long timePartition) {
+ return sequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public TsFileResourceList getUnsequenceListByTimePartition(long timePartition) {
+ return unsequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public Iterator<TsFileResource> getIterator(boolean sequence) {
+ readLock();
+ try {
+ return getTsFileList(sequence).iterator();
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void remove(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("remove");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (entry.getValue().contains(tsFileResource)) {
+ entry.getValue().remove(tsFileResource);
+ break;
+ }
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void removeAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ for (TsFileResource resource : tsFileResourceList) {
+ remove(resource, sequence);
+ }
+ }
+
+ /**
+ * insert tsFileResource to a target pos(targetPos = insertPos + 1) e.g. if insertPos = 0, then to
+ * the first, if insert Pos = 1, then to the second.
+ */
+ public void insert(TsFileResource tsFileResource, boolean sequence, int insertPos) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ TsFileResourceList tsFileResources =
+ selectedMap.computeIfAbsent(
+ tsFileResource.getTimePartition(), o -> new TsFileResourceList());
+ tsFileResources.set(insertPos, tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void add(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ selectedMap
+ .computeIfAbsent(tsFileResource.getTimePartition(), o -> new TsFileResourceList())
+ .add(tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void addRecover(TsFileResource tsFileResource, boolean sequence) {
Review comment:
```suggestion
public void addForRecover(TsFileResource tsFileResource, boolean sequence) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41594644)
Coverage increased (+1.0%) to 68.106% when pulling **e3ef41b0ab10e5a2246a215fa1b34f05fc0acb39 on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681046636
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1307,8 +1261,8 @@ private String getNewTsFileName(long timePartitionId) {
return getNewTsFileName(System.currentTimeMillis(), version, 0, 0);
}
- private String getNewTsFileName(long time, long version, int mergeCnt, int unSeqMergeCnt) {
- return TsFileResource.getNewTsFileName(time, version, mergeCnt, unSeqMergeCnt);
+ private String getNewTsFileName(long time, long version, int mergeCnt, int unseqMergeCnt) {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1449,9 +1403,9 @@ public void syncDeleteDataFiles() {
syncCloseAllWorkingTsFileProcessors();
// normally, mergingModification is just need to be closed by after a merge task is finished.
// we close it here just for IT test.
- if (this.tsFileManagement.mergingModification != null) {
- this.tsFileManagement.mergingModification.close();
- }
+ // if (this.tsFileResourceManager.mergingModification != null) {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1829,15 +1771,23 @@ public void delete(PartialPath path, long startTime, long endTime, long planInde
logDeletion(startTime, endTime, path);
Deletion deletion = new Deletion(path, MERGE_MOD_START_VERSION_NUM, startTime, endTime);
- if (tsFileManagement.mergingModification != null) {
- tsFileManagement.mergingModification.write(deletion);
- updatedModFiles.add(tsFileManagement.mergingModification);
- }
+ // if (tsFileResourceManager.mergingModification != null) {
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42197684)
Coverage increased (+0.03%) to 67.261% when pulling **b4785e34d554573544a0cee5663daf6001e57aeb on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42241264)
Coverage increased (+0.03%) to 67.255% when pulling **b8b3c234d4cb8beecc30025b75caa2c3354b7271 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42431817)
Coverage decreased (-0.01%) to 67.272% when pulling **113d863984a90987a0942d01b79a3bb22f931a8b on THUMarkLau:new_compaction** into **8ec3e12865699064958e36a37a50083d85e11f9c on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40837981)
Coverage decreased (-0.01%) to 67.888% when pulling **3ca922fd9aade2ad3e4fa1beedf40f3372299076 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43611775)
Coverage increased (+0.04%) to 67.483% when pulling **a0a64f8bc0ccac3bdf35f649d00066ad7f13baf9 on THUMarkLau:new_compaction** into **e4b7f64deb54b3fc186424cf969a68bff23a6fc7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40875517)
Coverage increased (+0.2%) to 68.08% when pulling **c3eb8a4cf60ea8945bde18bbb6e1eeb8ecf99d22 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43554797)
Coverage increased (+0.07%) to 67.818% when pulling **313760eda4195b81b959239701df80fcdc6667ed on THUMarkLau:new_compaction** into **c662a3e86de46aecc56236f0c2b693a2c479f38d on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40836981)
Coverage increased (+0.005%) to 67.907% when pulling **a096b2ca62283964a7de0a2062622c814b57bdd9 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43412690)
Coverage increased (+0.02%) to 67.765% when pulling **90fcf7cfe12c9ea72f18d8006f120de803d2b2d4 on THUMarkLau:new_compaction** into **4bae008971513bb65040031bcdb56f00bb9cd6d0 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43473275)
Coverage remained the same at 67.742% when pulling **393e0c41334eabf086831470bf477c96dab68612 on THUMarkLau:new_compaction** into **c662a3e86de46aecc56236f0c2b693a2c479f38d on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43722758)
Coverage decreased (-0.02%) to 67.235% when pulling **fc3ef51f493c61ac8f342f42d7c5ed1892f61e3b on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42244066)
Coverage increased (+0.03%) to 67.255% when pulling **56415dfb9cbe1340fa3c462344f3760184377038 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42201057)
Coverage increased (+0.03%) to 67.261% when pulling **c32d1cc5c6499e4535ba66b774038c096d759022 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42196741)
Coverage increased (+0.03%) to 67.26% when pulling **a092bae951de6d63150f9403aeb9cf674720a7f1 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42940936)
Coverage increased (+0.1%) to 67.541% when pulling **839ec9ae3920a52a063ed8edaa1ffc72e9d5f5d6 on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681046281
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/task/AbstractCompactionTask.java
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.compaction.task;
+
+import org.apache.iotdb.db.engine.compaction.CompactionScheduler;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.Callable;
+
+import static org.apache.iotdb.db.engine.compaction.CompactionScheduler.currentTaskNum;
+
+/**
+ * AbstractCompactionTask is the base class for all compaction task, it carries out the execution of
+ * compaction. AbstractCompactionTask uses a template method, it execute the abstract function
+ * <i>doCompaction</i> implemented by subclass, and decrease the currentTaskNum in
+ * CompactionScheduler when the <i>doCompaction</i> finish.
+ */
+public abstract class AbstractCompactionTask implements Callable<Void> {
+ private static final Logger LOGGER = LoggerFactory.getLogger(AbstractCompactionTask.class);
+ protected String fullStorageGroupName;
+ protected long timePartition;
+
+ public AbstractCompactionTask(String fullStorageGroupName, long timePartition) {
+ this.fullStorageGroupName = fullStorageGroupName;
+ this.timePartition = timePartition;
+ }
+
+ protected abstract void doCompaction() throws Exception;
+
+ @Override
+ public Void call() throws Exception {
+ try {
+ doCompaction();
+ } catch (Exception e) {
+ LOGGER.warn(e.getMessage(), e);
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41857432)
Coverage decreased (-0.01%) to 68.273% when pulling **d2dc97ac57e16179a368b7c4ba84e4f4facb9f2a on THUMarkLau:new_compaction** into **e3171bd7e0203923da0cfad3b9d8ebddd8211d70 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41711005)
Coverage increased (+0.003%) to 68.215% when pulling **c51ce1c87e9c8a0541dc4ed06ee010fa97bf5a41 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42209983)
Coverage increased (+0.07%) to 67.297% when pulling **3394ae319453ba1473d472b00f76de908bd819da on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681045463
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionTaskManager.java
##########
@@ -94,25 +89,8 @@ public void waitAndStop(long milliseconds) {
@TestOnly
public void waitAllCompactionFinish() {
if (pool != null) {
- File sgDir =
- FSFactoryProducer.getFSFactory()
- .getFile(
- FilePathUtils.regularizePath(
- IoTDBDescriptor.getInstance().getConfig().getSystemDir())
- + "storage_groups");
- File[] subDirList = sgDir.listFiles();
- if (subDirList != null) {
- for (File subDir : subDirList) {
- while (FSFactoryProducer.getFSFactory()
- .getFile(
- subDir.getAbsoluteFile()
- + File.separator
- + subDir.getName()
- + COMPACTION_LOG_NAME)
- .exists()) {
- // wait
- }
- }
+ while (CompactionScheduler.currentTaskNum.get() > 0) {
+ // wait
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42191287)
Coverage increased (+0.03%) to 67.258% when pulling **b41bf062825d2fe2be854cf935e6e611095d4f30 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40881245)
Coverage increased (+0.2%) to 68.076% when pulling **727bad675c65ef639defb7e551c1fac1215048d6 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42169601)
Coverage increased (+0.02%) to 67.249% when pulling **3aff8dd3d83d83f662091f203b372d218dbab011 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42941362)
Coverage increased (+0.06%) to 67.543% when pulling **f852341b5f672b314f11f4d46914c9f2a8937402 on THUMarkLau:new_compaction** into **e5d86c9d1d5de1934c77d2ea4f97f8cd6d669bb8 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42167583)
Coverage increased (+0.02%) to 67.245% when pulling **e740496691bc33542a3ab46cb9079295c99b6da1 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41668992)
Coverage increased (+0.005%) to 68.084% when pulling **52981823d4ee32f1807fcf2b20c364cf41b835a6 on THUMarkLau:new_compaction** into **72799f33215d5f6291544c354caf58c23a3dd04a on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41719140)
Coverage increased (+0.008%) to 68.221% when pulling **3f9232b99338b0e4ed3adbb171dc6744eb460c5a on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42233934)
Coverage increased (+0.03%) to 67.257% when pulling **a227a10d4cb2e75759ece2449dfc4ea644f3acf2 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681046078
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/CompactionUtils.java
##########
@@ -477,9 +484,79 @@ private static void modifyChunkMetaDataWithCache(
for (Modification modification : modifications) {
if (modification.getPath().matchFullPath(seriesPath)) {
seriesModifications.add(modification);
- usedModifications.add(modification);
}
}
modifyChunkMetaData(chunkMetadataList, seriesModifications);
}
+
+ private static boolean isFileListHasModifications(
+ Set<TsFileSequenceReader> readers, Map<String, List<Modification>> modificationCache) {
+ for (TsFileSequenceReader reader : readers) {
+ if (!getModifications(reader, modificationCache).isEmpty()) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static List<Modification> getModifications(
+ TsFileSequenceReader reader, Map<String, List<Modification>> modificationCache) {
+ return modificationCache.computeIfAbsent(
+ reader.getFileName(),
+ fileName ->
+ new LinkedList<>(
+ new ModificationFile(fileName + ModificationFile.FILE_SUFFIX).getModifications()));
+ }
+
+ public static ModificationFile getModificationByPath(String filePath) {
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41185411)
Coverage increased (+1.0%) to 68.121% when pulling **b08618bd1116892c0db2d1699a02d8d9a0c1fe4e on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41714690)
Coverage increased (+0.02%) to 68.229% when pulling **b13a741e261e84079592901b680940b5115cd6e4 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41833792)
Coverage increased (+0.1%) to 68.297% when pulling **712768e35f28bf2fc60ea8aa0089135a8a2aff31 on THUMarkLau:new_compaction** into **1d0b6ff0893a8ea1e43ad66d1b589770bc898933 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42878692)
Coverage decreased (-0.03%) to 67.392% when pulling **cac5572a7691a914989c21feb2976067cec13de4 on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43693072)
Coverage decreased (-0.004%) to 67.249% when pulling **d6378b4b349c2129448c57ff4fb8d8ad35b89d60 on THUMarkLau:new_compaction** into **516bf6588d18a58deb2088524cd042f1e938cc64 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41283919)
Coverage increased (+1.004%) to 68.151% when pulling **c2add8c7733190e03297e47da4add4a05e0f90bd on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/40842498)
Coverage decreased (-0.005%) to 67.896% when pulling **2296a3ef14c1647d5bf99be5f247afe3032c4a00 on THUMarkLau:new_compaction** into **5c6d11bd7e3f4b5e842bd3a71b262be5692e2ad6 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41503291)
Coverage increased (+1.0%) to 68.117% when pulling **c5246aba004f2e62e92f2c70941a2be0e4c1fb0c on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42780115)
Coverage increased (+0.08%) to 67.505% when pulling **5127a32c5d0b5f8b1cf7f94a1363cc315c98f4bd on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41354282)
Coverage increased (+1.02%) to 68.171% when pulling **e94ea5c35524afd222c4817a2103b84bb8e321ae on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42256923)
Coverage increased (+0.05%) to 67.274% when pulling **564bcff911cdcf307965b91495a0027a0ad06ff3 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681047429
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
+ throws IOException {
+ File tsFile = tsFileResource.getTsFile();
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFileResource.getTsFile().getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ tsFileResource.setFile(
+ new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX));
+ return tsFileResource;
+ }
+
+ public static File modifyTsFileNameUnseqMergCnt(File tsFile) throws IOException {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
Review comment:
remove
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
##########
@@ -438,7 +439,14 @@ public boolean resourceFileExists() {
public synchronized ModificationFile getModFile() {
if (modFile == null) {
- modFile = new ModificationFile(file.getPath() + ModificationFile.FILE_SUFFIX);
+ modFile = ModificationFile.getNormalMods(this);
+ }
+ return modFile;
+ }
+
+ public synchronized ModificationFile getCompactionModFile() {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
##########
@@ -438,7 +439,14 @@ public boolean resourceFileExists() {
public synchronized ModificationFile getModFile() {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java
##########
@@ -813,6 +813,11 @@ public void updatePlanIndexes(long planIndex) {
}
}
+ public static int getCompactionCount(String fileName) throws IOException {
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
+ throws IOException {
+ File tsFile = tsFileResource.getTsFile();
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFileResource.getTsFile().getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ tsFileResource.setFile(
+ new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX));
+ return tsFileResource;
+ }
+
+ public static File modifyTsFileNameUnseqMergCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File modifyTsFileNameMergeCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setMergeCnt(tsFileName.getMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File getInnerCompactionFileName(List<TsFileResource> tsFileResources)
+ throws IOException {
+ long minTime = Long.MAX_VALUE;
+ long minVersion = Long.MAX_VALUE;
+ long maxInnerMergeCount = Long.MIN_VALUE;
+ long maxCrossMergeCount = Long.MIN_VALUE;
+ for (TsFileResource resource : tsFileResources) {
+ TsFileName tsFileName = getTsFileName(resource.getTsFile().getName());
+ minTime = Math.min(tsFileName.time, minTime);
+ minVersion = Math.min(tsFileName.version, minVersion);
+ maxInnerMergeCount = Math.max(tsFileName.mergeCnt, maxInnerMergeCount);
+ maxCrossMergeCount = Math.max(tsFileName.unSeqMergeCnt, maxCrossMergeCount);
+ }
+ return new File(
+ tsFileResources.get(0).getTsFile().getParent(),
+ minTime
+ + FILE_NAME_SEPARATOR
+ + minVersion
+ + FILE_NAME_SEPARATOR
+ + (maxInnerMergeCount + 1)
+ + FILE_NAME_SEPARATOR
+ + maxCrossMergeCount
+ + TSFILE_SUFFIX);
+ }
+
+ public static class TsFileName {
+
+ private long time;
+ private long version;
+ private int mergeCnt;
Review comment:
fixed
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileNameGenerator.java
##########
@@ -0,0 +1,282 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBConstant;
+import org.apache.iotdb.db.conf.directories.DirectoryManager;
+import org.apache.iotdb.db.exception.DiskSpaceInsufficientException;
+import org.apache.iotdb.tsfile.common.constant.TsFileConstant;
+import org.apache.iotdb.tsfile.fileSystem.FSFactoryProducer;
+import org.apache.iotdb.tsfile.fileSystem.fsFactory.FSFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_MERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_SEPARATOR;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_TIME_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX;
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SUFFIX_VERSION_INDEX;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileNameGenerator {
+
+ private static FSFactory fsFactory = FSFactoryProducer.getFSFactory();
+
+ /**
+ * @param sequence whether the file is sequence
+ * @param logicalStorageGroup eg. "root.sg"
+ * @param virtualStorageGroup eg. "0"
+ * @param timePartitionId eg. 0
+ * @param time eg. 1623895965058
+ * @param version eg. 0
+ * @param innerSpaceCompactionCount the times of inner space compaction of this file
+ * @param crossSpaceCompactionCount the times of cross space compaction of this file
+ * @return a relative path of new tsfile, eg.
+ * "data/data/sequence/root.sg/0/0/1623895965058-0-0-0.tsfile"
+ */
+ public static String generateNewTsFilePath(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePath(
+ String tsFileDir,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount) {
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ public static String generateNewTsFilePatWithMkdir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId,
+ long time,
+ long version,
+ int innerSpaceCompactionCount,
+ int crossSpaceCompactionCount)
+ throws DiskSpaceInsufficientException, IOException {
+ String tsFileDir =
+ generateTsFileDir(sequence, logicalStorageGroup, virtualStorageGroup, timePartitionId);
+ fsFactory.getFile(tsFileDir).mkdirs();
+ return tsFileDir
+ + File.separator
+ + generateNewTsFileName(
+ time, version, innerSpaceCompactionCount, crossSpaceCompactionCount);
+ }
+
+ private static String generateTsFileDir(
+ boolean sequence,
+ String logicalStorageGroup,
+ String virtualStorageGroup,
+ long timePartitionId)
+ throws DiskSpaceInsufficientException {
+ DirectoryManager directoryManager = DirectoryManager.getInstance();
+ String baseDir =
+ sequence
+ ? directoryManager.getNextFolderForSequenceFile()
+ : directoryManager.getNextFolderForUnSequenceFile();
+ return baseDir
+ + File.separator
+ + logicalStorageGroup
+ + File.separator
+ + virtualStorageGroup
+ + File.separator
+ + timePartitionId;
+ }
+
+ public static String generateNewTsFileName(
+ long time, long version, int innerSpaceCompactionCount, int crossSpaceCompactionCount) {
+ return time
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + version
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + innerSpaceCompactionCount
+ + IoTDBConstant.FILE_NAME_SEPARATOR
+ + crossSpaceCompactionCount
+ + TsFileConstant.TSFILE_SUFFIX;
+ }
+
+ public static TsFileName getTsFileName(String fileName) throws IOException {
+ String[] fileNameParts =
+ fileName.split(FILE_NAME_SUFFIX_SEPARATOR)[FILE_NAME_SUFFIX_INDEX].split(
+ FILE_NAME_SEPARATOR);
+ if (fileNameParts.length != 4) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ try {
+ TsFileName tsFileName =
+ new TsFileName(
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_TIME_INDEX]),
+ Long.parseLong(fileNameParts[FILE_NAME_SUFFIX_VERSION_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_MERGECNT_INDEX]),
+ Integer.parseInt(fileNameParts[FILE_NAME_SUFFIX_UNSEQMERGECNT_INDEX]));
+ return tsFileName;
+ } catch (NumberFormatException e) {
+ throw new IOException("tsfile file name format is incorrect:" + fileName);
+ }
+ }
+
+ public static TsFileResource modifyTsFileNameUnseqMergCnt(TsFileResource tsFileResource)
+ throws IOException {
+ File tsFile = tsFileResource.getTsFile();
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFileResource.getTsFile().getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ tsFileResource.setFile(
+ new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX));
+ return tsFileResource;
+ }
+
+ public static File modifyTsFileNameUnseqMergCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setUnSeqMergeCnt(tsFileName.getUnSeqMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File modifyTsFileNameMergeCnt(File tsFile) throws IOException {
+ String path = tsFile.getParent();
+ TsFileName tsFileName = getTsFileName(tsFile.getName());
+ tsFileName.setMergeCnt(tsFileName.getMergeCnt() + 1);
+ return new File(
+ path,
+ tsFileName.time
+ + FILE_NAME_SEPARATOR
+ + tsFileName.version
+ + FILE_NAME_SEPARATOR
+ + tsFileName.mergeCnt
+ + FILE_NAME_SEPARATOR
+ + tsFileName.unSeqMergeCnt
+ + TSFILE_SUFFIX);
+ }
+
+ public static File getInnerCompactionFileName(List<TsFileResource> tsFileResources)
+ throws IOException {
+ long minTime = Long.MAX_VALUE;
+ long minVersion = Long.MAX_VALUE;
+ long maxInnerMergeCount = Long.MIN_VALUE;
+ long maxCrossMergeCount = Long.MIN_VALUE;
+ for (TsFileResource resource : tsFileResources) {
+ TsFileName tsFileName = getTsFileName(resource.getTsFile().getName());
+ minTime = Math.min(tsFileName.time, minTime);
+ minVersion = Math.min(tsFileName.version, minVersion);
+ maxInnerMergeCount = Math.max(tsFileName.mergeCnt, maxInnerMergeCount);
+ maxCrossMergeCount = Math.max(tsFileName.unSeqMergeCnt, maxCrossMergeCount);
+ }
+ return new File(
+ tsFileResources.get(0).getTsFile().getParent(),
+ minTime
+ + FILE_NAME_SEPARATOR
+ + minVersion
+ + FILE_NAME_SEPARATOR
+ + (maxInnerMergeCount + 1)
+ + FILE_NAME_SEPARATOR
+ + maxCrossMergeCount
+ + TSFILE_SUFFIX);
+ }
+
+ public static class TsFileName {
+
+ private long time;
+ private long version;
+ private int mergeCnt;
+ private int unSeqMergeCnt;
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41739089)
Coverage decreased (-0.001%) to 68.211% when pulling **ea1a8ce0e57b26fa2a8e4672ed47c903a73d5388 on THUMarkLau:new_compaction** into **d1729eafc6d9ba7833f728439c401d9af3409414 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41385839)
Coverage increased (+1.0%) to 68.136% when pulling **6ba30ebc48929309bb0eb6c860a4645af9fed811 on THUMarkLau:new_compaction** into **139c6bab7b295042636c2d3c36187b8935e13358 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681042647
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/cross/CrossCompactionStrategy.java
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.compaction.cross;
+
+import org.apache.iotdb.db.engine.compaction.cross.inplace.InplaceCompactionRecoverTask;
+import org.apache.iotdb.db.engine.compaction.cross.inplace.InplaceCompactionSelector;
+import org.apache.iotdb.db.engine.compaction.cross.inplace.InplaceCompactionTask;
+import org.apache.iotdb.db.engine.compaction.cross.inplace.manage.CrossSpaceMergeResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResourceList;
+
+import java.io.File;
+import java.util.List;
+
+public enum CrossCompactionStrategy {
+ INPLACE_COMPACTION;
+
+ public AbstractCrossSpaceCompactionTask getCompactionTask(
+ String logicalStorageGroupName,
+ String virtualStorageGroupName,
+ long timePartitionId,
+ CrossSpaceMergeResource mergeResource,
+ String storageGroupDir,
+ TsFileResourceList seqTsFileResourceList,
+ TsFileResourceList unSeqTsFileResourceList,
+ List<TsFileResource> selectedSeqTsFileResourceList,
+ List<TsFileResource> selectedUnSeqTsFileResourceList,
+ int concurrentMergeCount) {
+ switch (this) {
+ case INPLACE_COMPACTION:
+ default:
+ return new InplaceCompactionTask(
+ logicalStorageGroupName,
+ virtualStorageGroupName,
+ timePartitionId,
+ mergeResource,
+ storageGroupDir,
+ seqTsFileResourceList,
+ unSeqTsFileResourceList,
+ selectedSeqTsFileResourceList,
+ selectedUnSeqTsFileResourceList,
+ concurrentMergeCount);
+ }
+ }
+
+ public AbstractCrossSpaceCompactionTask getCompactionRecoverTask(
Review comment:
This change will result in a lot of redundant code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#discussion_r681048794
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResourceManager.java
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.exception.WriteLockFailedException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static org.apache.iotdb.db.conf.IoTDBConstant.FILE_NAME_SEPARATOR;
+import static org.apache.iotdb.tsfile.common.constant.TsFileConstant.TSFILE_SUFFIX;
+
+public class TsFileResourceManager {
+ private static final Logger LOGGER = LoggerFactory.getLogger(TsFileResourceManager.class);
+ private String storageGroupName;
+ private String virtualStorageGroup;
+ private String storageGroupDir;
+
+ /** Serialize queries, delete resource files, compaction cleanup files */
+ private final ReadWriteLock resourceListLock = new ReentrantReadWriteLock();
+
+ private String writeLockHolder;
+ // time partition -> double linked list of tsfiles
+ private Map<Long, TsFileResourceList> sequenceFiles = new TreeMap<>();
+ private Map<Long, TsFileResourceList> unsequenceFiles = new TreeMap<>();
+
+ private List<TsFileResource> sequenceRecoverTsFileResources = new ArrayList<>();
+ private List<TsFileResource> unsequenceRecoverTsFileResources = new ArrayList<>();
+
+ public TsFileResourceManager(
+ String storageGroupName, String virtualStorageGroup, String storageGroupDir) {
+ this.storageGroupName = storageGroupName;
+ this.storageGroupDir = storageGroupDir;
+ this.virtualStorageGroup = virtualStorageGroup;
+ }
+
+ public List<TsFileResource> getTsFileList(boolean sequence) {
+ readLock();
+ try {
+ List<TsFileResource> allResources = new ArrayList<>();
+ Map<Long, TsFileResourceList> chosenMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : chosenMap.entrySet()) {
+ allResources.addAll(entry.getValue().getArrayList());
+ }
+ return allResources;
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public TsFileResourceList getSequenceListByTimePartition(long timePartition) {
+ return sequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public TsFileResourceList getUnsequenceListByTimePartition(long timePartition) {
+ return unsequenceFiles.computeIfAbsent(timePartition, l -> new TsFileResourceList());
+ }
+
+ public Iterator<TsFileResource> getIterator(boolean sequence) {
+ readLock();
+ try {
+ return getTsFileList(sequence).iterator();
+ } finally {
+ readUnlock();
+ }
+ }
+
+ public void remove(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("remove");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ for (Map.Entry<Long, TsFileResourceList> entry : selectedMap.entrySet()) {
+ if (entry.getValue().contains(tsFileResource)) {
+ entry.getValue().remove(tsFileResource);
+ break;
+ }
+ }
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void removeAll(List<TsFileResource> tsFileResourceList, boolean sequence) {
+ for (TsFileResource resource : tsFileResourceList) {
+ remove(resource, sequence);
+ }
+ }
+
+ /**
+ * insert tsFileResource to a target pos(targetPos = insertPos + 1) e.g. if insertPos = 0, then to
+ * the first, if insert Pos = 1, then to the second.
+ */
+ public void insert(TsFileResource tsFileResource, boolean sequence, int insertPos) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ TsFileResourceList tsFileResources =
+ selectedMap.computeIfAbsent(
+ tsFileResource.getTimePartition(), o -> new TsFileResourceList());
+ tsFileResources.set(insertPos, tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void add(TsFileResource tsFileResource, boolean sequence) {
+ writeLock("add");
+ try {
+ Map<Long, TsFileResourceList> selectedMap = sequence ? sequenceFiles : unsequenceFiles;
+ selectedMap
+ .computeIfAbsent(tsFileResource.getTimePartition(), o -> new TsFileResourceList())
+ .add(tsFileResource);
+ } finally {
+ writeUnlock();
+ }
+ }
+
+ public void addRecover(TsFileResource tsFileResource, boolean sequence) {
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42880073)
Coverage decreased (-0.02%) to 67.396% when pulling **cd101a57694f96071b4599425229162e5d3c20b0 on THUMarkLau:new_compaction** into **601df56b76df7dbda720361a536ab66e492f1b25 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42230982)
Coverage increased (+0.05%) to 67.277% when pulling **6a1080f27ffb3e2e7b2055ca1c05dedffdabc039 on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/41852766)
Coverage increased (+0.008%) to 68.294% when pulling **58aebe44a831f07624cc386efc51ac7a623f1194 on THUMarkLau:new_compaction** into **e3171bd7e0203923da0cfad3b9d8ebddd8211d70 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43626914)
Coverage decreased (-0.2%) to 67.264% when pulling **3c4dc79f97c41006f4725b0060ff7a50ae13633d on THUMarkLau:new_compaction** into **e4b7f64deb54b3fc186424cf969a68bff23a6fc7 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE 3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/42191896)
Coverage increased (+0.04%) to 67.267% when pulling **895e246cf7ec50ec62dfcc32b14a9ee2ff82bb4f on THUMarkLau:new_compaction** into **7ea4f0d6d71bcb72b1e21a5684fdb79ae82707b4 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #3447: [ISSUE-3445] New compaction strategy and compaction scheduling strategy
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #3447:
URL: https://github.com/apache/iotdb/pull/3447#issuecomment-867311641
[](https://coveralls.io/builds/43405166)
Coverage decreased (-0.07%) to 67.726% when pulling **7883f87ac64543647548eb19e5b7ddb054ad827d on THUMarkLau:new_compaction** into **5ab9e4abeb117201620588874e77a24b2bf98671 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org