You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/03/07 08:23:09 UTC
[GitHub] [iotdb] choubenson opened a new pull request #5177: [IOTDB-2680]Fix too many open files exception
choubenson opened a new pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177
Compaction breaks at 0.12.5, then recover at 0.13, it will report "too many open files" exception.
Solutions:
1. Limit num of unseq files in cross space compaction.
2. Manage all tsfile readers by FileReaderManager.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#issuecomment-1060389671
[](https://coveralls.io/builds/47122617)
Coverage increased (+0.03%) to 67.799% when pulling **eeaed73fb279804ed318d9416ccd9645c5266344 on choubenson:fixTooManyOpenFiles** into **6d15dd0ebd11f170f6918d04fb94a5d8e6a56f38 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls commented on pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
coveralls commented on pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#issuecomment-1060389671
[](https://coveralls.io/builds/47118850)
Coverage increased (+0.04%) to 67.813% when pulling **32f7f41049c3595e3ddaa446e0d902474362533d on choubenson:fixTooManyOpenFiles** into **6d15dd0ebd11f170f6918d04fb94a5d8e6a56f38 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r820662391
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/MultiTsFileDeviceIterator.java
##########
@@ -72,6 +74,23 @@ public MultiTsFileDeviceIterator(List<TsFileResource> tsFileResources) throws IO
}
}
+ /** Used for cross space compaction. */
+ public MultiTsFileDeviceIterator(
Review comment:
If exception occurs here, end query may close some files which have not yet build reader.
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/compaction/utils/CompactionConfigRestorer.java
##########
@@ -58,7 +58,7 @@ public void restoreCompactionConfig() {
config.setTargetChunkPointNum(targetChunkPointNum);
config.setChunkSizeLowerBoundInCompaction(chunkSizeLowerBoundInCompaction);
config.setChunkPointNumLowerBoundInCompaction(chunkPointNumLowerBoundInCompaction);
- config.setMaxCompactionCandidateFileNum(maxCompactionCandidateFileNum);
+ config.setMaxInnerCompactionCandidateFileNum(maxCompactionCandidateFileNum);
config.setConcurrentCompactionThread(concurrentCompactionThread);
Review comment:
Add `config.setMaxCrossCompactionCandidateFileNum(1000)` here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#issuecomment-1060389671
[](https://coveralls.io/builds/47157760)
Coverage decreased (-0.008%) to 67.823% when pulling **ea91028e2a77d9e273ce13bb552ad46cd1fcbfd5 on choubenson:fixTooManyOpenFiles** into **766460048237acae1efa05f45f3d38481c01d83d on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r820613912
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/cross/rewrite/selector/RewriteCompactionFileSelector.java
##########
@@ -195,7 +198,9 @@ void select(boolean useTightBound) throws IOException {
}
private boolean updateSelectedFiles(long newCost, TsFileResource unseqFile) {
- if (totalCost + newCost < memoryBudget) {
+ if (selectedSeqFiles.size() + selectedUnseqFiles.size() + 1 + tmpSelectedSeqFiles.size()
+ < maxCrossCompactionFileNum
Review comment:
```suggestion
<= maxCrossCompactionFileNum
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
qiaojialin merged pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r820616190
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/MultiTsFileDeviceIterator.java
##########
@@ -72,6 +74,23 @@ public MultiTsFileDeviceIterator(List<TsFileResource> tsFileResources) throws IO
}
}
+ /** Used for cross space compaction. */
+ public MultiTsFileDeviceIterator(
+ List<TsFileResource> seqResources, List<TsFileResource> unseqResources) throws IOException {
+ for (TsFileResource tsFileResource : seqResources) {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(tsFileResource.getTsFilePath(), true);
+ readerMap.put(tsFileResource, reader);
+ deviceIteratorMap.put(tsFileResource, reader.getAllDevicesIteratorWithIsAligned());
+ }
+ for (TsFileResource tsFileResource : unseqResources) {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(tsFileResource.getTsFilePath(), true);
+ readerMap.put(tsFileResource, reader);
+ deviceIteratorMap.put(tsFileResource, reader.getAllDevicesIteratorWithIsAligned());
+ }
Review comment:
catch throwable and close all readers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] coveralls edited a comment on pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
coveralls edited a comment on pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#issuecomment-1060389671
[](https://coveralls.io/builds/47153695)
Coverage increased (+0.04%) to 67.812% when pulling **f0be704e1382c6ef68e17cf8519c0e986f92f806 on choubenson:fixTooManyOpenFiles** into **6d15dd0ebd11f170f6918d04fb94a5d8e6a56f38 on apache:master**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r821427682
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/MultiTsFileDeviceIterator.java
##########
@@ -72,6 +74,23 @@ public MultiTsFileDeviceIterator(List<TsFileResource> tsFileResources) throws IO
}
}
+ /** Used for cross space compaction. */
+ public MultiTsFileDeviceIterator(
Review comment:
Get it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
choubenson commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r821284321
##########
File path: server/src/test/java/org/apache/iotdb/db/engine/compaction/utils/CompactionConfigRestorer.java
##########
@@ -58,7 +58,7 @@ public void restoreCompactionConfig() {
config.setTargetChunkPointNum(targetChunkPointNum);
config.setChunkSizeLowerBoundInCompaction(chunkSizeLowerBoundInCompaction);
config.setChunkPointNumLowerBoundInCompaction(chunkPointNumLowerBoundInCompaction);
- config.setMaxCompactionCandidateFileNum(maxCompactionCandidateFileNum);
+ config.setMaxInnerCompactionCandidateFileNum(maxCompactionCandidateFileNum);
config.setConcurrentCompactionThread(concurrentCompactionThread);
Review comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
choubenson commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r821323566
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/MultiTsFileDeviceIterator.java
##########
@@ -72,6 +74,23 @@ public MultiTsFileDeviceIterator(List<TsFileResource> tsFileResources) throws IO
}
}
+ /** Used for cross space compaction. */
+ public MultiTsFileDeviceIterator(
+ List<TsFileResource> seqResources, List<TsFileResource> unseqResources) throws IOException {
+ for (TsFileResource tsFileResource : seqResources) {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(tsFileResource.getTsFilePath(), true);
+ readerMap.put(tsFileResource, reader);
+ deviceIteratorMap.put(tsFileResource, reader.getAllDevicesIteratorWithIsAligned());
+ }
+ for (TsFileResource tsFileResource : unseqResources) {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance().get(tsFileResource.getTsFilePath(), true);
+ readerMap.put(tsFileResource, reader);
+ deviceIteratorMap.put(tsFileResource, reader.getAllDevicesIteratorWithIsAligned());
+ }
Review comment:
If the reader is closed in this method, there will be problems. Query f1, merge f1 and f2, and throw an exception when creating f2 reader in compaction. If f1 is closed in this method, the query will report an error. And f2 reader will be closed in end query.
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/inner/utils/MultiTsFileDeviceIterator.java
##########
@@ -72,6 +74,23 @@ public MultiTsFileDeviceIterator(List<TsFileResource> tsFileResources) throws IO
}
}
+ /** Used for cross space compaction. */
+ public MultiTsFileDeviceIterator(
Review comment:
If the reader is closed in this method, there will be problems. Query f1, merge f1 and f2, and throw an exception when creating f2 reader in compaction. If f1 is closed in this method, the query will report an error. And f2 reader will be closed in end query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a change in pull request #5177: [IOTDB-2680]Fix too many open files exception
Posted by GitBox <gi...@apache.org>.
choubenson commented on a change in pull request #5177:
URL: https://github.com/apache/iotdb/pull/5177#discussion_r821272386
##########
File path: server/src/main/java/org/apache/iotdb/db/engine/compaction/cross/rewrite/selector/RewriteCompactionFileSelector.java
##########
@@ -195,7 +198,9 @@ void select(boolean useTightBound) throws IOException {
}
private boolean updateSelectedFiles(long newCost, TsFileResource unseqFile) {
- if (totalCost + newCost < memoryBudget) {
+ if (selectedSeqFiles.size() + selectedUnseqFiles.size() + 1 + tmpSelectedSeqFiles.size()
+ < maxCrossCompactionFileNum
Review comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org