You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by GitBox <gi...@apache.org> on 2022/04/05 07:38:12 UTC
[GitHub] [iotdb] choubenson opened a new pull request, #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
choubenson opened a new pull request, #5415:
URL: https://github.com/apache/iotdb/pull/5415
The cross space comapction execution performance of 0.13.0 is slower than 0.12.5.
Need to speed up.
We achieve parallel reading and writing at the measurement level by adding threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r846597579
##########
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java:
##########
@@ -458,6 +458,10 @@ public void writePlanIndices() throws IOException {
out.flush();
}
+ public void truncate(long truncateSize) throws IOException {
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842650642
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java:
##########
@@ -83,21 +81,25 @@ public void endChunkGroup() throws IOException {
}
@Override
- public void endMeasurement() throws IOException {
- writeRateLimit(chunkWriter.estimateMaxSeriesMemSize());
- chunkWriter.writeToFileWriter(fileWriterList.get(seqFileIndex));
- chunkWriter = null;
- seqFileIndex = 0;
+ public void endMeasurement(int subTaskId) throws IOException {
+ writeRateLimit(chunkWriterMap.get(subTaskId).estimateMaxSeriesMemSize());
+ synchronized (fileWriterList.get(seqFileIndexMap.get(subTaskId))) {
+ chunkWriterMap
+ .get(subTaskId)
+ .writeToFileWriter(fileWriterList.get(seqFileIndexMap.get(subTaskId)));
+ }
+ // chunkWriterMap.get(subTaskId)=null;
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842650906
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/InnerSpaceCompactionWriter.java:
##########
@@ -80,7 +79,13 @@ public void close() throws IOException {
if (fileWriter != null && fileWriter.canWrite()) {
fileWriter.close();
}
- chunkWriter = null;
+ // chunkWriter = null;
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842497941
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -170,59 +178,57 @@ private static void compactNonAlignedSeries(
AbstractCompactionWriter compactionWriter,
QueryContext queryContext,
QueryDataSource queryDataSource)
- throws MetadataException, IOException {
- boolean hasStartChunkGroup = false;
+ throws IOException {
MultiTsFileDeviceIterator.MeasurementIterator measurementIterator =
deviceIterator.iterateNotAlignedSeries(device, false);
Set<String> allMeasurements = measurementIterator.getAllMeasurements();
+ int subTaskNums = Math.min(allMeasurements.size(), subTaskNum);
+
+ // assign all measurements to different sub tasks
+ Set<String>[] measurementsForEachSubTask = new HashSet[subTaskNums];
+ int idx = 0;
for (String measurement : allMeasurements) {
- List<IMeasurementSchema> measurementSchemas = new ArrayList<>();
- try {
- if (IoTDBDescriptor.getInstance().getConfig().isEnableIDTable()) {
- measurementSchemas.add(IDTableManager.getInstance().getSeriesSchema(device, measurement));
- } else {
- measurementSchemas.add(
- IoTDB.schemaProcessor.getSeriesSchema(new PartialPath(device, measurement)));
- }
- } catch (PathNotExistException e) {
- logger.info("A deleted path is skipped: {}", e.getMessage());
- continue;
+ if (measurementsForEachSubTask[idx % subTaskNums] == null) {
+ measurementsForEachSubTask[idx % subTaskNums] = new HashSet<String>();
}
+ measurementsForEachSubTask[idx++ % subTaskNums].add(measurement);
+ }
- IBatchReader dataBatchReader =
- constructReader(
- device,
- Collections.singletonList(measurement),
- measurementSchemas,
- allMeasurements,
- queryContext,
- queryDataSource,
- false);
+ // construct sub tasks and start compacting measurements in parallel
+ List<Future<Void>> futures = new ArrayList<>();
+ compactionWriter.startChunkGroup(device, false);
+ for (int i = 0; i < subTaskNums; i++) {
+ futures.add(
+ CompactionTaskManager.getInstance()
+ .submitSubTask(
+ new SubCompactionTask(
+ device,
+ measurementsForEachSubTask[i],
+ queryContext,
+ queryDataSource,
+ compactionWriter,
+ i)));
+ }
- if (dataBatchReader.hasNextBatch()) {
- if (!hasStartChunkGroup) {
- // chunkgroup is serialized only when at least one timeseries under this device has
- // data
- compactionWriter.startChunkGroup(device, false);
- hasStartChunkGroup = true;
- }
- compactionWriter.startMeasurement(measurementSchemas);
- writeWithReader(compactionWriter, dataBatchReader);
- compactionWriter.endMeasurement();
+ // wait for all sub tasks finish
+ for (int i = 0; i < subTaskNums; i++) {
+ try {
+ futures.get(i).get();
+ } catch (InterruptedException | ExecutionException e) {
+ logger.error("SubCompactionTask meet errors ", e);
Review Comment:
Thread.interrepted().
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java:
##########
@@ -83,21 +81,25 @@ public void endChunkGroup() throws IOException {
}
@Override
- public void endMeasurement() throws IOException {
- writeRateLimit(chunkWriter.estimateMaxSeriesMemSize());
- chunkWriter.writeToFileWriter(fileWriterList.get(seqFileIndex));
- chunkWriter = null;
- seqFileIndex = 0;
+ public void endMeasurement(int subTaskId) throws IOException {
+ writeRateLimit(chunkWriterMap.get(subTaskId).estimateMaxSeriesMemSize());
+ synchronized (fileWriterList.get(seqFileIndexMap.get(subTaskId))) {
+ chunkWriterMap
+ .get(subTaskId)
+ .writeToFileWriter(fileWriterList.get(seqFileIndexMap.get(subTaskId)));
+ }
+ // chunkWriterMap.get(subTaskId)=null;
Review Comment:
remove
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java:
##########
@@ -50,34 +52,34 @@
MetricConfigDescriptor.getInstance().getMetricConfig().getEnableMetric();
// point count in current measurment, which is used to check size
Review Comment:
update doc.
data point in each subTask?
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java:
##########
@@ -36,36 +36,34 @@
// source tsfiles
private List<TsFileResource> seqTsFileResources;
- private int seqFileIndex;
+ // subTaskId -> seqFileIndex
+ private Map<Integer, Integer> seqFileIndexMap;
Review Comment:
Use Integer[]
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java:
##########
@@ -50,34 +52,34 @@
MetricConfigDescriptor.getInstance().getMetricConfig().getEnableMetric();
// point count in current measurment, which is used to check size
- private int measurementPointCount;
+ private Map<Integer, Integer> measurementPointCountMap = new ConcurrentHashMap<>();
Review Comment:
Use AtomicInteger[]
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -268,6 +274,27 @@ private static AbstractCompactionWriter getCompactionWriter(
}
}
+ private static void updateDeviceStartTimeAndEndTime(
+ List<TsFileResource> targetResources, AbstractCompactionWriter compactionWriter) {
+ List<TsFileIOWriter> targetFileWriters = compactionWriter.getFileIOWriter();
+ for (int i = 0; i < targetFileWriters.size(); i++) {
+ TsFileIOWriter fileIOWriter = targetFileWriters.get(i);
+ TsFileResource fileResource = targetResources.get(i);
+ // skip the target file that has been deleted
Review Comment:
why deleted, add comment
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/InnerSpaceCompactionWriter.java:
##########
@@ -80,7 +79,13 @@ public void close() throws IOException {
if (fileWriter != null && fileWriter.canWrite()) {
fileWriter.close();
}
- chunkWriter = null;
+ // chunkWriter = null;
Review Comment:
remove
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/InnerSpaceCompactionWriter.java:
##########
@@ -50,17 +48,18 @@ public void endChunkGroup() throws IOException {
}
@Override
- public void endMeasurement() throws IOException {
- writeRateLimit(chunkWriter.estimateMaxSeriesMemSize());
- chunkWriter.writeToFileWriter(fileWriter);
- chunkWriter = null;
+ public void endMeasurement(int subTaskId) throws IOException {
+ writeRateLimit(chunkWriterMap.get(subTaskId).estimateMaxSeriesMemSize());
+ synchronized (fileWriter) {
+ chunkWriterMap.get(subTaskId).writeToFileWriter(fileWriter);
+ }
+ // chunkWriter = null;
Review Comment:
remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] THUMarkLau commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
THUMarkLau commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842675942
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -170,59 +178,58 @@ private static void compactNonAlignedSeries(
AbstractCompactionWriter compactionWriter,
QueryContext queryContext,
QueryDataSource queryDataSource)
- throws MetadataException, IOException {
- boolean hasStartChunkGroup = false;
+ throws IOException {
MultiTsFileDeviceIterator.MeasurementIterator measurementIterator =
deviceIterator.iterateNotAlignedSeries(device, false);
Set<String> allMeasurements = measurementIterator.getAllMeasurements();
+ int subTaskNums = Math.min(allMeasurements.size(), subTaskNum);
+
+ // assign all measurements to different sub tasks
+ Set<String>[] measurementsForEachSubTask = new HashSet[subTaskNums];
+ int idx = 0;
for (String measurement : allMeasurements) {
- List<IMeasurementSchema> measurementSchemas = new ArrayList<>();
- try {
- if (IoTDBDescriptor.getInstance().getConfig().isEnableIDTable()) {
- measurementSchemas.add(IDTableManager.getInstance().getSeriesSchema(device, measurement));
- } else {
- measurementSchemas.add(
- IoTDB.schemaProcessor.getSeriesSchema(new PartialPath(device, measurement)));
- }
- } catch (PathNotExistException e) {
- logger.info("A deleted path is skipped: {}", e.getMessage());
- continue;
+ if (measurementsForEachSubTask[idx % subTaskNums] == null) {
Review Comment:
Is it necessary to hash here? If the hash is not uniform, then the effect of multithreading could not be worse. Why not just use a concurrency stack or queue, each sub-thread get a measurement to compact when it is available.
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -170,59 +178,58 @@ private static void compactNonAlignedSeries(
AbstractCompactionWriter compactionWriter,
QueryContext queryContext,
QueryDataSource queryDataSource)
- throws MetadataException, IOException {
- boolean hasStartChunkGroup = false;
+ throws IOException {
MultiTsFileDeviceIterator.MeasurementIterator measurementIterator =
deviceIterator.iterateNotAlignedSeries(device, false);
Set<String> allMeasurements = measurementIterator.getAllMeasurements();
+ int subTaskNums = Math.min(allMeasurements.size(), subTaskNum);
+
+ // assign all measurements to different sub tasks
+ Set<String>[] measurementsForEachSubTask = new HashSet[subTaskNums];
+ int idx = 0;
for (String measurement : allMeasurements) {
- List<IMeasurementSchema> measurementSchemas = new ArrayList<>();
- try {
- if (IoTDBDescriptor.getInstance().getConfig().isEnableIDTable()) {
- measurementSchemas.add(IDTableManager.getInstance().getSeriesSchema(device, measurement));
- } else {
- measurementSchemas.add(
- IoTDB.schemaProcessor.getSeriesSchema(new PartialPath(device, measurement)));
- }
- } catch (PathNotExistException e) {
- logger.info("A deleted path is skipped: {}", e.getMessage());
- continue;
+ if (measurementsForEachSubTask[idx % subTaskNums] == null) {
+ measurementsForEachSubTask[idx % subTaskNums] = new HashSet<String>();
}
+ measurementsForEachSubTask[idx++ % subTaskNums].add(measurement);
+ }
- IBatchReader dataBatchReader =
- constructReader(
- device,
- Collections.singletonList(measurement),
- measurementSchemas,
- allMeasurements,
- queryContext,
- queryDataSource,
- false);
+ // construct sub tasks and start compacting measurements in parallel
+ List<Future<Void>> futures = new ArrayList<>();
+ compactionWriter.startChunkGroup(device, false);
+ for (int i = 0; i < subTaskNums; i++) {
+ futures.add(
+ CompactionTaskManager.getInstance()
+ .submitSubTask(
+ new SubCompactionTask(
+ device,
+ measurementsForEachSubTask[i],
+ queryContext,
+ queryDataSource,
+ compactionWriter,
+ i)));
+ }
- if (dataBatchReader.hasNextBatch()) {
- if (!hasStartChunkGroup) {
- // chunkgroup is serialized only when at least one timeseries under this device has
- // data
- compactionWriter.startChunkGroup(device, false);
- hasStartChunkGroup = true;
- }
- compactionWriter.startMeasurement(measurementSchemas);
- writeWithReader(compactionWriter, dataBatchReader);
- compactionWriter.endMeasurement();
+ // wait for all sub tasks finish
+ for (int i = 0; i < subTaskNums; i++) {
+ try {
+ futures.get(i).get();
+ } catch (InterruptedException | ExecutionException e) {
+ logger.error("SubCompactionTask meet errors ", e);
+ Thread.interrupted();
+ throw new IOException(e);
Review Comment:
Throw a InterruptedException instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842593853
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java:
##########
@@ -36,36 +36,34 @@
// source tsfiles
private List<TsFileResource> seqTsFileResources;
- private int seqFileIndex;
+ // subTaskId -> seqFileIndex
+ private Map<Integer, Integer> seqFileIndexMap;
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842651100
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/InnerSpaceCompactionWriter.java:
##########
@@ -50,17 +48,18 @@ public void endChunkGroup() throws IOException {
}
@Override
- public void endMeasurement() throws IOException {
- writeRateLimit(chunkWriter.estimateMaxSeriesMemSize());
- chunkWriter.writeToFileWriter(fileWriter);
- chunkWriter = null;
+ public void endMeasurement(int subTaskId) throws IOException {
+ writeRateLimit(chunkWriterMap.get(subTaskId).estimateMaxSeriesMemSize());
+ synchronized (fileWriter) {
+ chunkWriterMap.get(subTaskId).writeToFileWriter(fileWriter);
+ }
+ // chunkWriter = null;
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842650238
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java:
##########
@@ -50,34 +52,34 @@
MetricConfigDescriptor.getInstance().getMetricConfig().getEnableMetric();
// point count in current measurment, which is used to check size
- private int measurementPointCount;
+ private Map<Integer, Integer> measurementPointCountMap = new ConcurrentHashMap<>();
Review Comment:
Resolved
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java:
##########
@@ -50,34 +52,34 @@
MetricConfigDescriptor.getInstance().getMetricConfig().getEnableMetric();
// point count in current measurment, which is used to check size
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r843736461
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -170,59 +178,58 @@ private static void compactNonAlignedSeries(
AbstractCompactionWriter compactionWriter,
QueryContext queryContext,
QueryDataSource queryDataSource)
- throws MetadataException, IOException {
- boolean hasStartChunkGroup = false;
+ throws IOException {
MultiTsFileDeviceIterator.MeasurementIterator measurementIterator =
deviceIterator.iterateNotAlignedSeries(device, false);
Set<String> allMeasurements = measurementIterator.getAllMeasurements();
+ int subTaskNums = Math.min(allMeasurements.size(), subTaskNum);
+
+ // assign all measurements to different sub tasks
+ Set<String>[] measurementsForEachSubTask = new HashSet[subTaskNums];
+ int idx = 0;
for (String measurement : allMeasurements) {
- List<IMeasurementSchema> measurementSchemas = new ArrayList<>();
- try {
- if (IoTDBDescriptor.getInstance().getConfig().isEnableIDTable()) {
- measurementSchemas.add(IDTableManager.getInstance().getSeriesSchema(device, measurement));
- } else {
- measurementSchemas.add(
- IoTDB.schemaProcessor.getSeriesSchema(new PartialPath(device, measurement)));
- }
- } catch (PathNotExistException e) {
- logger.info("A deleted path is skipped: {}", e.getMessage());
- continue;
+ if (measurementsForEachSubTask[idx % subTaskNums] == null) {
+ measurementsForEachSubTask[idx % subTaskNums] = new HashSet<String>();
}
+ measurementsForEachSubTask[idx++ % subTaskNums].add(measurement);
+ }
- IBatchReader dataBatchReader =
- constructReader(
- device,
- Collections.singletonList(measurement),
- measurementSchemas,
- allMeasurements,
- queryContext,
- queryDataSource,
- false);
+ // construct sub tasks and start compacting measurements in parallel
+ List<Future<Void>> futures = new ArrayList<>();
+ compactionWriter.startChunkGroup(device, false);
+ for (int i = 0; i < subTaskNums; i++) {
+ futures.add(
+ CompactionTaskManager.getInstance()
+ .submitSubTask(
+ new SubCompactionTask(
+ device,
+ measurementsForEachSubTask[i],
+ queryContext,
+ queryDataSource,
+ compactionWriter,
+ i)));
+ }
- if (dataBatchReader.hasNextBatch()) {
- if (!hasStartChunkGroup) {
- // chunkgroup is serialized only when at least one timeseries under this device has
- // data
- compactionWriter.startChunkGroup(device, false);
- hasStartChunkGroup = true;
- }
- compactionWriter.startMeasurement(measurementSchemas);
- writeWithReader(compactionWriter, dataBatchReader);
- compactionWriter.endMeasurement();
+ // wait for all sub tasks finish
+ for (int i = 0; i < subTaskNums; i++) {
+ try {
+ futures.get(i).get();
+ } catch (InterruptedException | ExecutionException e) {
+ logger.error("SubCompactionTask meet errors ", e);
+ Thread.interrupted();
+ throw new IOException(e);
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r846580617
##########
tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java:
##########
@@ -458,6 +458,10 @@ public void writePlanIndices() throws IOException {
out.flush();
}
+ public void truncate(long truncateSize) throws IOException {
Review Comment:
```suggestion
public void truncate(long offset) throws IOException {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r844925483
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -170,59 +178,58 @@ private static void compactNonAlignedSeries(
AbstractCompactionWriter compactionWriter,
QueryContext queryContext,
QueryDataSource queryDataSource)
- throws MetadataException, IOException {
- boolean hasStartChunkGroup = false;
+ throws IOException {
MultiTsFileDeviceIterator.MeasurementIterator measurementIterator =
deviceIterator.iterateNotAlignedSeries(device, false);
Set<String> allMeasurements = measurementIterator.getAllMeasurements();
+ int subTaskNums = Math.min(allMeasurements.size(), subTaskNum);
+
+ // assign all measurements to different sub tasks
+ Set<String>[] measurementsForEachSubTask = new HashSet[subTaskNums];
+ int idx = 0;
for (String measurement : allMeasurements) {
- List<IMeasurementSchema> measurementSchemas = new ArrayList<>();
- try {
- if (IoTDBDescriptor.getInstance().getConfig().isEnableIDTable()) {
- measurementSchemas.add(IDTableManager.getInstance().getSeriesSchema(device, measurement));
- } else {
- measurementSchemas.add(
- IoTDB.schemaProcessor.getSeriesSchema(new PartialPath(device, measurement)));
- }
- } catch (PathNotExistException e) {
- logger.info("A deleted path is skipped: {}", e.getMessage());
- continue;
+ if (measurementsForEachSubTask[idx % subTaskNums] == null) {
Review Comment:
Each sub task has its own hashset, so this won't happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin merged pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
qiaojialin merged PR #5415:
URL: https://github.com/apache/iotdb/pull/5415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] choubenson commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
choubenson commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r842659290
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/CompactionUtils.java:
##########
@@ -268,6 +274,27 @@ private static AbstractCompactionWriter getCompactionWriter(
}
}
+ private static void updateDeviceStartTimeAndEndTime(
+ List<TsFileResource> targetResources, AbstractCompactionWriter compactionWriter) {
+ List<TsFileIOWriter> targetFileWriters = compactionWriter.getFileIOWriter();
+ for (int i = 0; i < targetFileWriters.size(); i++) {
+ TsFileIOWriter fileIOWriter = targetFileWriters.get(i);
+ TsFileResource fileResource = targetResources.get(i);
+ // skip the target file that has been deleted
Review Comment:
Resolved
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/AbstractCompactionWriter.java:
##########
@@ -50,34 +52,34 @@
MetricConfigDescriptor.getInstance().getMetricConfig().getEnableMetric();
// point count in current measurment, which is used to check size
- private int measurementPointCount;
+ private Map<Integer, Integer> measurementPointCountMap = new ConcurrentHashMap<>();
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] qiaojialin commented on a diff in pull request #5415: [IOTDB-2807]Speed up the cross space compaction by multi-threads
Posted by GitBox <gi...@apache.org>.
qiaojialin commented on code in PR #5415:
URL: https://github.com/apache/iotdb/pull/5415#discussion_r845743529
##########
server/src/main/java/org/apache/iotdb/db/engine/compaction/writer/CrossSpaceCompactionWriter.java:
##########
@@ -169,10 +181,34 @@ private void checkIsDeviceExistAndGetDeviceEndTime() throws IOException {
}
}
- private void checkAndMayStartChunkGroup() throws IOException {
- if (!hasTargetFileStartChunkGroup[seqFileIndex]) {
- fileWriterList.get(seqFileIndex).startChunkGroup(deviceId);
- hasTargetFileStartChunkGroup[seqFileIndex] = true;
+ private void checkAndMayStartChunkGroup(int subTaskId) throws IOException {
+ int fileIndex = seqFileIndexArray[subTaskId];
+ if (hasTargetFileStartChunkGroup[fileIndex].compareAndSet(0, 1)) {
+ fileWriterList.get(fileIndex).startChunkGroup(deviceId);
+ hasTargetFileStartChunkGroup[fileIndex].set(2);
+ }
+ }
+
+ private void flushChunkToFileWriter(int subTaskId) throws IOException {
+ int fileIndex = seqFileIndexArray[subTaskId];
+ writeRateLimit(chunkWriterMap.get(subTaskId).estimateMaxSeriesMemSize());
+ while (hasTargetFileStartChunkGroup[fileIndex].get() == 1) {
+ // wait until the target file has finished flushing chunk group header
Review Comment:
wait and notifyall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org