You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2020/03/23 10:25:21 UTC
[GitHub] [incubator-iotdb] jt2594838 opened a new pull request #935:
Integrate data file version recording with time partitioning
jt2594838 opened a new pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935
Previously the data file version recording is not integrated with time partitioning, so versions of different partitions may be mixed.
This PR will make file versions of different partitions be managed separately using a map.
Some other minor fixes and extractions are also included to improve the code quality.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396863074
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
+ }
+ }
+ return insertPos;
+ }
+
+ /**
+ * Compare each device in the two files to find the time relation of them.
+ * @param fileA
+ * @param fileB
+ * @return -1 if fileA is totally older than fileB (A < B)
+ * 0 if fileA is partially older than fileB and partially newer than fileB (A X B)
+ * 1 if fileA is totally newer than fileB (B < A)
+ */
+ private int compareTsFileDevices(TsFileResource fileA, TsFileResource fileB) {
+ boolean hasPre = false, hasSubsequence = false;
+ for (String device : fileA.getStartTimeMap().keySet()) {
+ if (!fileB.getStartTimeMap().containsKey(device)) {
+ continue;
+ }
+ long startTimeA = fileA.getStartTimeMap().get(device);
+ long endTimeA = fileA.getEndTimeMap().get(device);
+ long startTimeB = fileB.getStartTimeMap().get(device);
+ long endTimeB = fileB.getEndTimeMap().get(device);
+ if (startTimeA > endTimeB) {
+ // A's data of the device is later than to the B's data
+ hasSubsequence = true;
+ } else if (startTimeB > endTimeA) {
+ // A's data of the device is previous to the B's data
+ hasPre = true;
+ } else {
+ // the two files overlap in the device
+ return 0;
+ }
+ }
+ if (hasPre && hasSubsequence) {
+ // some devices are newer but some devices are older, the two files overlap in general
+ return 0;
+ }
+ if (!hasPre && hasSubsequence) {
+ // all devices in B are newer than those in A
+ return -1;
+ }
+ // all devices in B are older than those in A
Review comment:
All devices in A are older than those in B.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603601329
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396863018
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
+ }
+ }
+ return insertPos;
+ }
+
+ /**
+ * Compare each device in the two files to find the time relation of them.
+ * @param fileA
+ * @param fileB
+ * @return -1 if fileA is totally older than fileB (A < B)
+ * 0 if fileA is partially older than fileB and partially newer than fileB (A X B)
+ * 1 if fileA is totally newer than fileB (B < A)
+ */
+ private int compareTsFileDevices(TsFileResource fileA, TsFileResource fileB) {
+ boolean hasPre = false, hasSubsequence = false;
+ for (String device : fileA.getStartTimeMap().keySet()) {
+ if (!fileB.getStartTimeMap().containsKey(device)) {
+ continue;
+ }
+ long startTimeA = fileA.getStartTimeMap().get(device);
+ long endTimeA = fileA.getEndTimeMap().get(device);
+ long startTimeB = fileB.getStartTimeMap().get(device);
+ long endTimeB = fileB.getEndTimeMap().get(device);
+ if (startTimeA > endTimeB) {
+ // A's data of the device is later than to the B's data
+ hasSubsequence = true;
+ } else if (startTimeB > endTimeA) {
+ // A's data of the device is previous to the B's data
+ hasPre = true;
+ } else {
+ // the two files overlap in the device
+ return 0;
+ }
+ }
+ if (hasPre && hasSubsequence) {
+ // some devices are newer but some devices are older, the two files overlap in general
+ return 0;
+ }
+ if (!hasPre && hasSubsequence) {
+ // all devices in B are newer than those in A
Review comment:
All devices in A are newer than those in B.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396889714
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1559,61 +1601,24 @@ public void loadNewTsFileForSync(TsFileResource newTsFileResource)
public void loadNewTsFile(TsFileResource newTsFileResource)
throws TsFileProcessorException {
File tsfileToBeInserted = newTsFileResource.getFile();
+
writeLock();
mergeLock.writeLock().lock();
try {
- boolean isOverlap = false;
- int preIndex = -1, subsequentIndex = sequenceFileTreeSet.size();
-
- List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- // check new tsfile
- outer:
- for (int i = 0; i < sequenceList.size(); i++) {
- if (sequenceList.get(i).getFile().getName().equals(tsfileToBeInserted.getName())) {
- return;
- }
- if (i == sequenceList.size() - 1 && sequenceList.get(i).getEndTimeMap().isEmpty()) {
- continue;
- }
- boolean hasPre = false, hasSubsequence = false;
- for (String device : newTsFileResource.getStartTimeMap().keySet()) {
- if (sequenceList.get(i).getStartTimeMap().containsKey(device)) {
- long startTime1 = sequenceList.get(i).getStartTimeMap().get(device);
- long endTime1 = sequenceList.get(i).getEndTimeMap().get(device);
- long startTime2 = newTsFileResource.getStartTimeMap().get(device);
- long endTime2 = newTsFileResource.getEndTimeMap().get(device);
- if (startTime1 > endTime2) {
- hasSubsequence = true;
- } else if (startTime2 > endTime1) {
- hasPre = true;
- } else {
- isOverlap = true;
- break outer;
- }
- }
- }
- if (hasPre && hasSubsequence) {
- isOverlap = true;
- break;
- }
- if (!hasPre && hasSubsequence) {
- subsequentIndex = i;
- break;
- }
- if (hasPre) {
- preIndex = i;
- }
+ int insertPos = findInsertionPosition(newTsFileResource);
+ if (insertPos == POS_ALREADY_EXIST) {
+ return;
}
// loading tsfile by type
- if (isOverlap) {
+ if (insertPos == POS_OVERLAP) {
loadTsFileByType(LoadTsFileType.LOAD_UNSEQUENCE, tsfileToBeInserted, newTsFileResource);
} else {
// check whether the file name needs to be renamed.
- if (subsequentIndex != sequenceFileTreeSet.size() || preIndex != -1) {
- String newFileName = getFileNameForLoadingFile(tsfileToBeInserted.getName(), preIndex,
- subsequentIndex, getTimePartitionFromTsFileResource(newTsFileResource));
+ if (insertPos != sequenceFileTreeSet.size() - 1 || insertPos != -1) {
Review comment:
Changed. But I doubt the quality of them, as the file being inserted at the head or the tail does not necessarily indicate the set is empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603656886
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r397266579
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1655,7 +1655,7 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
*/
private int findInsertionPosition(TsFileResource newTsFileResource) {
File tsfileToBeInserted = newTsFileResource.getFile();
- long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParentFile().getName());
Review comment:
The parent file of the file may not be its time range. It's better to get time range from the time.
Such as:
`load /data/1584445970309-101-1.tsfile`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603651523
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603651523
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396868261
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
Review comment:
`return i;`
It needs to add in the cur pos and move all newer files backward.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603699309
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-602998893
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396864697
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
Review comment:
case 1:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603636129
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603656886
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-602520427
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396888012
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
+ }
+ }
+ return insertPos;
+ }
+
+ /**
+ * Compare each device in the two files to find the time relation of them.
+ * @param fileA
+ * @param fileB
+ * @return -1 if fileA is totally older than fileB (A < B)
+ * 0 if fileA is partially older than fileB and partially newer than fileB (A X B)
+ * 1 if fileA is totally newer than fileB (B < A)
+ */
+ private int compareTsFileDevices(TsFileResource fileA, TsFileResource fileB) {
+ boolean hasPre = false, hasSubsequence = false;
+ for (String device : fileA.getStartTimeMap().keySet()) {
+ if (!fileB.getStartTimeMap().containsKey(device)) {
+ continue;
+ }
+ long startTimeA = fileA.getStartTimeMap().get(device);
+ long endTimeA = fileA.getEndTimeMap().get(device);
+ long startTimeB = fileB.getStartTimeMap().get(device);
+ long endTimeB = fileB.getEndTimeMap().get(device);
+ if (startTimeA > endTimeB) {
+ // A's data of the device is later than to the B's data
+ hasSubsequence = true;
+ } else if (startTimeB > endTimeA) {
+ // A's data of the device is previous to the B's data
+ hasPre = true;
+ } else {
+ // the two files overlap in the device
+ return 0;
+ }
+ }
+ if (hasPre && hasSubsequence) {
+ // some devices are newer but some devices are older, the two files overlap in general
+ return 0;
+ }
+ if (!hasPre && hasSubsequence) {
+ // all devices in B are newer than those in A
+ return -1;
+ }
+ // all devices in B are older than those in A
Review comment:
See above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396870747
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1559,61 +1601,24 @@ public void loadNewTsFileForSync(TsFileResource newTsFileResource)
public void loadNewTsFile(TsFileResource newTsFileResource)
throws TsFileProcessorException {
File tsfileToBeInserted = newTsFileResource.getFile();
+
writeLock();
mergeLock.writeLock().lock();
try {
- boolean isOverlap = false;
- int preIndex = -1, subsequentIndex = sequenceFileTreeSet.size();
-
- List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- // check new tsfile
- outer:
- for (int i = 0; i < sequenceList.size(); i++) {
- if (sequenceList.get(i).getFile().getName().equals(tsfileToBeInserted.getName())) {
- return;
- }
- if (i == sequenceList.size() - 1 && sequenceList.get(i).getEndTimeMap().isEmpty()) {
- continue;
- }
- boolean hasPre = false, hasSubsequence = false;
- for (String device : newTsFileResource.getStartTimeMap().keySet()) {
- if (sequenceList.get(i).getStartTimeMap().containsKey(device)) {
- long startTime1 = sequenceList.get(i).getStartTimeMap().get(device);
- long endTime1 = sequenceList.get(i).getEndTimeMap().get(device);
- long startTime2 = newTsFileResource.getStartTimeMap().get(device);
- long endTime2 = newTsFileResource.getEndTimeMap().get(device);
- if (startTime1 > endTime2) {
- hasSubsequence = true;
- } else if (startTime2 > endTime1) {
- hasPre = true;
- } else {
- isOverlap = true;
- break outer;
- }
- }
- }
- if (hasPre && hasSubsequence) {
- isOverlap = true;
- break;
- }
- if (!hasPre && hasSubsequence) {
- subsequentIndex = i;
- break;
- }
- if (hasPre) {
- preIndex = i;
- }
+ int insertPos = findInsertionPosition(newTsFileResource);
+ if (insertPos == POS_ALREADY_EXIST) {
+ return;
}
// loading tsfile by type
- if (isOverlap) {
+ if (insertPos == POS_OVERLAP) {
loadTsFileByType(LoadTsFileType.LOAD_UNSEQUENCE, tsfileToBeInserted, newTsFileResource);
} else {
// check whether the file name needs to be renamed.
- if (subsequentIndex != sequenceFileTreeSet.size() || preIndex != -1) {
- String newFileName = getFileNameForLoadingFile(tsfileToBeInserted.getName(), preIndex,
- subsequentIndex, getTimePartitionFromTsFileResource(newTsFileResource));
+ if (insertPos != sequenceFileTreeSet.size() - 1 || insertPos != -1) {
+ String newFileName = getFileNameForLoadingFile(tsfileToBeInserted.getName(), insertPos,
+ getTimePartitionFromTsFileResource(newTsFileResource));
Review comment:
Just a note: currently, `insertPos ` is equals to `subsequentIndex `.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603636129
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin merged pull request #935: Integrate
data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
qiaojialin merged pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-602998893
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396887993
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
+ }
+ }
+ return insertPos;
+ }
+
+ /**
+ * Compare each device in the two files to find the time relation of them.
+ * @param fileA
+ * @param fileB
+ * @return -1 if fileA is totally older than fileB (A < B)
+ * 0 if fileA is partially older than fileB and partially newer than fileB (A X B)
+ * 1 if fileA is totally newer than fileB (B < A)
+ */
+ private int compareTsFileDevices(TsFileResource fileA, TsFileResource fileB) {
+ boolean hasPre = false, hasSubsequence = false;
+ for (String device : fileA.getStartTimeMap().keySet()) {
+ if (!fileB.getStartTimeMap().containsKey(device)) {
+ continue;
+ }
+ long startTimeA = fileA.getStartTimeMap().get(device);
+ long endTimeA = fileA.getEndTimeMap().get(device);
+ long startTimeB = fileB.getStartTimeMap().get(device);
+ long endTimeB = fileB.getEndTimeMap().get(device);
+ if (startTimeA > endTimeB) {
+ // A's data of the device is later than to the B's data
+ hasSubsequence = true;
+ } else if (startTimeB > endTimeA) {
+ // A's data of the device is previous to the B's data
+ hasPre = true;
+ } else {
+ // the two files overlap in the device
+ return 0;
+ }
+ }
+ if (hasPre && hasSubsequence) {
+ // some devices are newer but some devices are older, the two files overlap in general
+ return 0;
+ }
+ if (!hasPre && hasSubsequence) {
+ // all devices in B are newer than those in A
Review comment:
I have changed the assignments of hasPre and hasSubsequence, now hasPre means "B has previous data to A", so this should remain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r397568389
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1655,7 +1655,7 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
*/
private int findInsertionPosition(TsFileResource newTsFileResource) {
File tsfileToBeInserted = newTsFileResource.getFile();
- long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParentFile().getName());
Review comment:
Yes, I agree with you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396868604
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
Review comment:
`return i + 1;`
It needs to be inserted after the older files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396870041
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1702,24 +1796,26 @@ private void removeFullyOverlapFiles(TsFileResource resource, Iterator<TsFileRes
* version number is the version number in the tsfile with a larger timestamp.
*
* @param tsfileName origin tsfile name
+ * @param insertIndex the new file will be inserted between the files [insertIndex, insertIndex
+ * + 1]
* @return appropriate filename
*/
- private String getFileNameForLoadingFile(String tsfileName, int preIndex, int subsequentIndex,
+ private String getFileNameForLoadingFile(String tsfileName, int insertIndex,
long timePartitionId) {
long currentTsFileTime = Long
.parseLong(tsfileName.split(IoTDBConstant.TSFILE_NAME_SEPARATOR)[0]);
long preTime;
List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- if (preIndex == -1) {
+ if (insertIndex == -1) {
preTime = 0L;
} else {
- String preName = sequenceList.get(preIndex).getFile().getName();
+ String preName = sequenceList.get(insertIndex).getFile().getName();
preTime = Long.parseLong(preName.split(IoTDBConstant.TSFILE_NAME_SEPARATOR)[0]);
}
- if (subsequentIndex == sequenceFileTreeSet.size()) {
+ if (insertIndex == sequenceFileTreeSet.size() - 1) {
Review comment:
`insertIndex == sequenceFileTreeSet.size()`
Logic:
If it needs to be inserted to be the last in the list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603662282
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396890295
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1702,24 +1796,26 @@ private void removeFullyOverlapFiles(TsFileResource resource, Iterator<TsFileRes
* version number is the version number in the tsfile with a larger timestamp.
*
* @param tsfileName origin tsfile name
+ * @param insertIndex the new file will be inserted between the files [insertIndex, insertIndex
+ * + 1]
* @return appropriate filename
*/
- private String getFileNameForLoadingFile(String tsfileName, int preIndex, int subsequentIndex,
+ private String getFileNameForLoadingFile(String tsfileName, int insertIndex,
long timePartitionId) {
long currentTsFileTime = Long
.parseLong(tsfileName.split(IoTDBConstant.TSFILE_NAME_SEPARATOR)[0]);
long preTime;
List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- if (preIndex == -1) {
+ if (insertIndex == -1) {
preTime = 0L;
} else {
- String preName = sequenceList.get(preIndex).getFile().getName();
+ String preName = sequenceList.get(insertIndex).getFile().getName();
preTime = Long.parseLong(preName.split(IoTDBConstant.TSFILE_NAME_SEPARATOR)[0]);
}
- if (subsequentIndex == sequenceFileTreeSet.size()) {
+ if (insertIndex == sequenceFileTreeSet.size() - 1) {
Review comment:
You have misunderstood the meaning of insertIndex, please read the Javadoc. For example, if the insertIndex is -1, it means the new file will be inserted between [-1, 0], not that the new file will be inserted at -1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396870041
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1702,24 +1796,26 @@ private void removeFullyOverlapFiles(TsFileResource resource, Iterator<TsFileRes
* version number is the version number in the tsfile with a larger timestamp.
*
* @param tsfileName origin tsfile name
+ * @param insertIndex the new file will be inserted between the files [insertIndex, insertIndex
+ * + 1]
* @return appropriate filename
*/
- private String getFileNameForLoadingFile(String tsfileName, int preIndex, int subsequentIndex,
+ private String getFileNameForLoadingFile(String tsfileName, int insertIndex,
long timePartitionId) {
long currentTsFileTime = Long
.parseLong(tsfileName.split(IoTDBConstant.TSFILE_NAME_SEPARATOR)[0]);
long preTime;
List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- if (preIndex == -1) {
+ if (insertIndex == -1) {
preTime = 0L;
} else {
- String preName = sequenceList.get(preIndex).getFile().getName();
+ String preName = sequenceList.get(insertIndex).getFile().getName();
preTime = Long.parseLong(preName.split(IoTDBConstant.TSFILE_NAME_SEPARATOR)[0]);
}
- if (subsequentIndex == sequenceFileTreeSet.size()) {
+ if (insertIndex == sequenceFileTreeSet.size() - 1) {
Review comment:
`insertIndex == sequenceFileTreeSet.size()`
Logic:
If it needs to be inserted to be the last in the list.
Just a note: currently, `insertPos ` is equals to `subsequentIndex `.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396888507
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
+ }
+ }
+ return insertPos;
+ }
+
+ /**
+ * Compare each device in the two files to find the time relation of them.
+ * @param fileA
+ * @param fileB
+ * @return -1 if fileA is totally older than fileB (A < B)
+ * 0 if fileA is partially older than fileB and partially newer than fileB (A X B)
+ * 1 if fileA is totally newer than fileB (B < A)
+ */
+ private int compareTsFileDevices(TsFileResource fileA, TsFileResource fileB) {
Review comment:
Yes, I mistakenly reversed the condition and now it is fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603662282
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] removed a comment on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] removed a comment on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603005479
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396888366
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
Review comment:
I believe it should be -1, which means data in the new file has smaller timestamps than the local file, so we can insert the new file before the local file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396888795
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
Review comment:
As I have explained in the javadoc that the file will be inserted between [i, i +1], I think it does not really matter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603005479
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396869149
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1559,61 +1601,24 @@ public void loadNewTsFileForSync(TsFileResource newTsFileResource)
public void loadNewTsFile(TsFileResource newTsFileResource)
throws TsFileProcessorException {
File tsfileToBeInserted = newTsFileResource.getFile();
+
writeLock();
mergeLock.writeLock().lock();
try {
- boolean isOverlap = false;
- int preIndex = -1, subsequentIndex = sequenceFileTreeSet.size();
-
- List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- // check new tsfile
- outer:
- for (int i = 0; i < sequenceList.size(); i++) {
- if (sequenceList.get(i).getFile().getName().equals(tsfileToBeInserted.getName())) {
- return;
- }
- if (i == sequenceList.size() - 1 && sequenceList.get(i).getEndTimeMap().isEmpty()) {
- continue;
- }
- boolean hasPre = false, hasSubsequence = false;
- for (String device : newTsFileResource.getStartTimeMap().keySet()) {
- if (sequenceList.get(i).getStartTimeMap().containsKey(device)) {
- long startTime1 = sequenceList.get(i).getStartTimeMap().get(device);
- long endTime1 = sequenceList.get(i).getEndTimeMap().get(device);
- long startTime2 = newTsFileResource.getStartTimeMap().get(device);
- long endTime2 = newTsFileResource.getEndTimeMap().get(device);
- if (startTime1 > endTime2) {
- hasSubsequence = true;
- } else if (startTime2 > endTime1) {
- hasPre = true;
- } else {
- isOverlap = true;
- break outer;
- }
- }
- }
- if (hasPre && hasSubsequence) {
- isOverlap = true;
- break;
- }
- if (!hasPre && hasSubsequence) {
- subsequentIndex = i;
- break;
- }
- if (hasPre) {
- preIndex = i;
- }
+ int insertPos = findInsertionPosition(newTsFileResource);
+ if (insertPos == POS_ALREADY_EXIST) {
+ return;
}
// loading tsfile by type
- if (isOverlap) {
+ if (insertPos == POS_OVERLAP) {
loadTsFileByType(LoadTsFileType.LOAD_UNSEQUENCE, tsfileToBeInserted, newTsFileResource);
} else {
// check whether the file name needs to be renamed.
- if (subsequentIndex != sequenceFileTreeSet.size() || preIndex != -1) {
- String newFileName = getFileNameForLoadingFile(tsfileToBeInserted.getName(), preIndex,
- subsequentIndex, getTimePartitionFromTsFileResource(newTsFileResource));
+ if (insertPos != sequenceFileTreeSet.size() - 1 || insertPos != -1) {
Review comment:
You can use `if(! sequenceFileTreeSet.isEmpty())`.
The origin `if (subsequentIndex != sequenceFileTreeSet.size() || preIndex != -1)` is equals to above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r397565649
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1655,7 +1655,7 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
*/
private int findInsertionPosition(TsFileResource newTsFileResource) {
File tsfileToBeInserted = newTsFileResource.getFile();
- long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParentFile().getName());
Review comment:
Fixed. But there is still one thing remaining, what if the file belongs to more than one partition? Currently, I will report an error and a delicate solution will be left as a future job.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-602520427
Kudos, SonarCloud Quality Gate passed!
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] sonarcloud[bot] commented on issue #935:
Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on issue #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#issuecomment-603601329
SonarCloud Quality Gate failed.
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug.png' alt='Bug' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=BUG)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability.png' alt='Vulnerability' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A.png' alt='A' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=VULNERABILITY) (and [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot.png' alt='Security Hotspot' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=SECURITY_HOTSPOT) to review)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell.png' alt='Code Smell' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/B.png' alt='B' width='16' height='16' />](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_incubator-iotdb&pullRequest=935&resolved=false&types=CODE_SMELL)
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo.png' alt='No Coverage information' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935) No Coverage information
[<img src='https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3.png' alt='0.0%' width='16' height='16' />](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache_incubator-iotdb&pullRequest=935&metric=new_duplicated_lines_density&view=list)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396870747
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1559,61 +1601,24 @@ public void loadNewTsFileForSync(TsFileResource newTsFileResource)
public void loadNewTsFile(TsFileResource newTsFileResource)
throws TsFileProcessorException {
File tsfileToBeInserted = newTsFileResource.getFile();
+
writeLock();
mergeLock.writeLock().lock();
try {
- boolean isOverlap = false;
- int preIndex = -1, subsequentIndex = sequenceFileTreeSet.size();
-
- List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
- // check new tsfile
- outer:
- for (int i = 0; i < sequenceList.size(); i++) {
- if (sequenceList.get(i).getFile().getName().equals(tsfileToBeInserted.getName())) {
- return;
- }
- if (i == sequenceList.size() - 1 && sequenceList.get(i).getEndTimeMap().isEmpty()) {
- continue;
- }
- boolean hasPre = false, hasSubsequence = false;
- for (String device : newTsFileResource.getStartTimeMap().keySet()) {
- if (sequenceList.get(i).getStartTimeMap().containsKey(device)) {
- long startTime1 = sequenceList.get(i).getStartTimeMap().get(device);
- long endTime1 = sequenceList.get(i).getEndTimeMap().get(device);
- long startTime2 = newTsFileResource.getStartTimeMap().get(device);
- long endTime2 = newTsFileResource.getEndTimeMap().get(device);
- if (startTime1 > endTime2) {
- hasSubsequence = true;
- } else if (startTime2 > endTime1) {
- hasPre = true;
- } else {
- isOverlap = true;
- break outer;
- }
- }
- }
- if (hasPre && hasSubsequence) {
- isOverlap = true;
- break;
- }
- if (!hasPre && hasSubsequence) {
- subsequentIndex = i;
- break;
- }
- if (hasPre) {
- preIndex = i;
- }
+ int insertPos = findInsertionPosition(newTsFileResource);
+ if (insertPos == POS_ALREADY_EXIST) {
+ return;
}
// loading tsfile by type
- if (isOverlap) {
+ if (insertPos == POS_OVERLAP) {
loadTsFileByType(LoadTsFileType.LOAD_UNSEQUENCE, tsfileToBeInserted, newTsFileResource);
} else {
// check whether the file name needs to be renamed.
- if (subsequentIndex != sequenceFileTreeSet.size() || preIndex != -1) {
- String newFileName = getFileNameForLoadingFile(tsfileToBeInserted.getName(), preIndex,
- subsequentIndex, getTimePartitionFromTsFileResource(newTsFileResource));
+ if (insertPos != sequenceFileTreeSet.size() - 1 || insertPos != -1) {
+ String newFileName = getFileNameForLoadingFile(tsfileToBeInserted.getName(), insertPos,
+ getTimePartitionFromTsFileResource(newTsFileResource));
Review comment:
Just a note: currently, `insertPos ` is equals to `subsequentIndex `.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
jt2594838 commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396888894
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
Review comment:
See above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request
#935: Integrate data file version recording with time partitioning
Posted by GitBox <gi...@apache.org>.
fanhualta commented on a change in pull request #935: Integrate data file version recording with time partitioning
URL: https://github.com/apache/incubator-iotdb/pull/935#discussion_r396867571
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -1638,12 +1646,98 @@ public void loadNewTsFile(TsFileResource newTsFileResource)
}
}
+ /**
+ * Find the position of "newTsFileResource" in the sequence files if it can be inserted into them.
+ * @param newTsFileResource
+ * @return POS_ALREADY_EXIST(-2) if some file has the same name as the one to be inserted
+ * POS_OVERLAP(-3) if some file overlaps the new file
+ * an insertion position i >= -1 if the new file can be inserted between [i, i+1]
+ */
+ private int findInsertionPosition(TsFileResource newTsFileResource) {
+ File tsfileToBeInserted = newTsFileResource.getFile();
+ long newFilePartitionId = Long.parseLong(tsfileToBeInserted.getParent());
+ int insertPos = -1;
+
+ List<TsFileResource> sequenceList = new ArrayList<>(sequenceFileTreeSet);
+ // find the position where the new file should be inserted
+ for (int i = 0; i < sequenceList.size(); i++) {
+ TsFileResource localFile = sequenceList.get(i);
+ if (localFile.getFile().getName().equals(tsfileToBeInserted.getName())) {
+ return POS_ALREADY_EXIST;
+ }
+ long localPartitionId = Long.parseLong(localFile.getFile().getParent());
+ if (i == sequenceList.size() - 1 && localFile.getEndTimeMap().isEmpty()
+ || newFilePartitionId != localPartitionId) {
+ // skip files that are not in the partition as the new file and the last empty file
+ continue;
+ }
+
+ int fileComparison = compareTsFileDevices(newTsFileResource, localFile);
+ switch (fileComparison) {
+ case 0:
+ // some devices are newer but some devices are older, the two files overlap in general
+ return POS_OVERLAP;
+ case -1:
+ // all devices in the local file are newer than the new file, the new file can be
+ // inserted before the new file
+ return i - 1;
+ default:
+ // all devices in the local file are older than the new file, proceed to the next file
+ insertPos = i;
+ }
+ }
+ return insertPos;
+ }
+
+ /**
+ * Compare each device in the two files to find the time relation of them.
+ * @param fileA
+ * @param fileB
+ * @return -1 if fileA is totally older than fileB (A < B)
+ * 0 if fileA is partially older than fileB and partially newer than fileB (A X B)
+ * 1 if fileA is totally newer than fileB (B < A)
+ */
+ private int compareTsFileDevices(TsFileResource fileA, TsFileResource fileB) {
Review comment:
I think this method needs to be checked carefully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services