You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/02/05 08:13:52 UTC
[PR] Implement table & tree disk usage statistics [iotdb]
shuwenwei opened a new pull request, #17169:
URL: https://github.com/apache/iotdb/pull/17169
## Description
Implement table & tree disk usage statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2928308839
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java:
##########
@@ -388,10 +414,31 @@ public class InformationSchema {
schemaTables.put(SERVICES, servicesTable);
}
+ static {
+ columnsThatSupportPushDownPredicate.put(
+ TABLE_DISK_USAGE,
+ ImmutableSet.of(
+ ColumnHeaderConstant.DATABASE,
+ ColumnHeaderConstant.TABLE_NAME_TABLE_MODEL,
+ ColumnHeaderConstant.DATA_NODE_ID_TABLE_MODEL,
+ ColumnHeaderConstant.REGION_ID_TABLE_MODEL,
+ ColumnHeaderConstant.TIME_PARTITION_TABLE_MODEL));
Review Comment:
columnsThatSupportPushDownPredicate for information_schema.table_disk_usage uses ColumnHeaderConstant.DATABASE ("Database"), but the table schema defines this column as lowercase "database" (via `ColumnHeaderConstant.DATABASE.toLowerCase(...)`). This case mismatch prevents predicate pushdown on the database column. Use the same lowercase column name here (or a dedicated table-model constant) so the pushdown checker matches SymbolReference names correctly.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/process/CollectNode.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.planner.plan.node.process;
+
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeType;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanVisitor;
+
+import com.google.common.base.Objects;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+public class CollectNode extends MultiChildProcessNode {
+
+ private final List<String> outputColumnNames;
+
+ public CollectNode(PlanNodeId id, List<String> outputColumnNames) {
+ super(id);
+ this.outputColumnNames = outputColumnNames;
+ }
+
+ public CollectNode(PlanNodeId id, List<PlanNode> children, List<String> outputColumnNames) {
+ super(id, children);
+ this.outputColumnNames = outputColumnNames;
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitCollect(this, context);
+ }
+
+ @Override
+ public PlanNode clone() {
+ return new CollectNode(id, outputColumnNames);
+ }
+
+ @Override
+ public PlanNodeType getType() {
+ return PlanNodeType.TREE_COLLECT;
+ }
+
+ @Override
+ public List<String> getOutputColumnNames() {
+ return outputColumnNames;
+ }
+
+ @Override
+ public PlanNode replaceChildren(List<PlanNode> newChildren) {
+ checkArgument(children.size() == newChildren.size(), "wrong number of new children");
+ return new CollectNode(id, newChildren, outputColumnNames);
+ }
+
+ @Override
+ protected void serializeAttributes(ByteBuffer byteBuffer) {
+ PlanNodeType.TREE_COLLECT.serialize(byteBuffer);
+ ReadWriteIOUtils.write(outputColumnNames.size(), byteBuffer);
+ outputColumnNames.forEach(column -> ReadWriteIOUtils.write(column, byteBuffer));
+ }
+
+ @Override
+ protected void serializeAttributes(DataOutputStream stream) throws IOException {
+ PlanNodeType.TREE_COLLECT.serialize(stream);
+ ReadWriteIOUtils.write(outputColumnNames.size(), stream);
+ for (String outputColumnName : outputColumnNames) {
+ ReadWriteIOUtils.write(outputColumnName, stream);
+ }
+ }
+
+ public static CollectNode deserialize(ByteBuffer byteBuffer) {
+ int size = ReadWriteIOUtils.readInt(byteBuffer);
+ List<String> outputColumnNames = new ArrayList<>(size);
+ while (size-- > 0) {
+ outputColumnNames.add(ReadWriteIOUtils.readString(byteBuffer));
+ }
+ PlanNodeId planNodeId = PlanNodeId.deserialize(byteBuffer);
+ return new CollectNode(planNodeId, outputColumnNames);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ if (!super.equals(o)) return false;
+ return false;
+ }
Review Comment:
CollectNode.equals() currently returns false for all non-identical instances (it never compares any fields and ends with `return false`). This breaks the equals/hashCode contract and can cause incorrect plan comparisons/caching and brittle tests. Implement equals() to compare relevant state (at least superclass + outputColumnNames, and typically children via super.equals).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2938673635
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
Review Comment:
Thread.currentThread().interrupt(); and stop state should be handled differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2938609532
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
Review Comment:
Operation operation = queue.poll(5, TimeUnit.SECONDS);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-4053475238
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3857795449
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2938036636
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
Review Comment:
should not return, you should continue, if it's not in stop state, you can refer to `AbstractDriverThread`
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/AbstractTableSizeCacheWriter.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.db.storageengine.StorageEngine;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+public abstract class AbstractTableSizeCacheWriter {
+ protected static final Logger logger =
+ LoggerFactory.getLogger(AbstractTableSizeCacheWriter.class);
+ public static final String TEMP_CACHE_FILE_SUBFIX = ".tmp";
+ protected final int regionId;
+ protected long previousCompactionTimestamp = System.currentTimeMillis();
+ protected long lastWriteTimestamp = System.currentTimeMillis();
+ protected int currentIndexFileVersion = 0;
+ protected final File dir;
+
+ public AbstractTableSizeCacheWriter(String database, int regionId) {
+ this.regionId = regionId;
+ this.dir = StorageEngine.getDataRegionSystemDir(database, regionId + "");
+ }
+
+ protected void failedToRecover(Exception e) {
+ TableDiskUsageCache.getInstance().failedToRecover(e);
+ }
+
+ protected void deleteOldVersionFiles(int maxVersion, String prefix, List<File> files) {
+ for (File file : files) {
+ try {
+ int version = Integer.parseInt(file.getName().substring(prefix.length()));
+ if (version != maxVersion) {
+ Files.deleteIfExists(file.toPath());
+ }
+ } catch (Exception e) {
+ logger.warn(
+ "Failed to delete old version table size cache file {}", file.getAbsolutePath());
+ }
+ }
+ }
+
+ public void closeIfIdle() {
+ if (System.currentTimeMillis() - lastWriteTimestamp >= TimeUnit.MINUTES.toMillis(1)) {
Review Comment:
```suggestion
if (System.currentTimeMillis() - lastWriteTimestamp >= TimeUnit.MINUTES.toMillis(60)) {
```
1min is too short
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/tsfile/TsFileTableDiskUsageCacheWriter.java:
##########
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.utils.FileUtils;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+
+import org.apache.tsfile.utils.Pair;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+
+public class TsFileTableDiskUsageCacheWriter extends AbstractTableSizeCacheWriter {
+ public static final String TSFILE_CACHE_KEY_FILENAME_PREFIX = "TableSizeKeyFile_";
+ public static final String TSFILE_CACHE_VALUE_FILENAME_PREFIX = "TableSizeValueFile_";
+ public static final int KEY_FILE_OFFSET_RECORD_LENGTH = 5 * Long.BYTES + 1;
+ public static final int KEY_FILE_REDIRECT_RECORD_LENGTH = 7 * Long.BYTES + 1;
Review Comment:
never used
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/InformationSchemaContentSupplierFactory.java:
##########
@@ -1165,7 +1215,297 @@ public boolean hasNext() {
}
}
- private abstract static class TsBlockSupplier implements Iterator<TsBlock> {
+ private static class TableDiskUsageSupplier implements IInformationSchemaContentSupplier {
+ private final List<TSDataType> dataTypes;
+ private final Map<String, List<TTableInfo>> databaseTableInfoMap;
+ private final Filter pushDownFilter;
+ private final PaginationController paginationController;
+ private final OperatorContext operatorContext;
+
+ private DataRegion currentDataRegion;
+ private boolean currentDatabaseOnlyHasOneTable;
+
+ private TableDiskUsageCacheReader currentDataRegionCacheReader;
+ private DataRegionTableSizeQueryContext currentDataRegionTableSizeQueryContext;
+
+ private final StorageEngineTimePartitionIterator dataRegionIterator;
+
+ private long prepareCacheReaderCostInNS = 0;
+ private long loadObjectFileCostInNS = 0;
+ private long prepareCachedTsFileIDCostInNS = 0;
+ private long checkAllFilesInTsFileManagerCostInNS = 0;
+ private long readTsFileCacheValueFilesCostInNS = 0;
+
+ private TableDiskUsageSupplier(
+ final List<TSDataType> dataTypes,
+ final UserEntity userEntity,
+ final Filter pushDownFilter,
+ final PaginationController paginationController,
+ final OperatorContext operatorContext,
+ final List<Integer> regionsForCurrentSubTask)
+ throws TException, ClientManagerException {
+ this.dataTypes = dataTypes;
+ this.pushDownFilter = pushDownFilter;
+ this.paginationController = paginationController;
+ this.operatorContext = operatorContext;
+ AuthorityChecker.getAccessControl().checkUserGlobalSysPrivilege(userEntity);
+ this.databaseTableInfoMap =
+ operatorContext.getInstanceContext().getDataNodeQueryContext().getDatabaseTableInfoMap();
+ Set<Integer> regions = new HashSet<>(regionsForCurrentSubTask);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.REGIONS_OF_CURRENT_SUB_TASK, regionsForCurrentSubTask.toString());
+ this.dataRegionIterator =
+ new StorageEngineTimePartitionIterator(
+ Optional.of(
+ dataRegion -> {
+ List<TTableInfo> tTableInfos =
+ databaseTableInfoMap.get(dataRegion.getDatabaseName());
+ if (tTableInfos == null || tTableInfos.isEmpty()) {
+ return false;
+ }
+ if (!regions.contains(dataRegion.getDataRegionId())) {
+ return false;
+ }
+ currentDataRegionTableSizeQueryContext =
+ new DataRegionTableSizeQueryContext(
+ false, operatorContext.getInstanceContext());
+ return true;
+ }),
+ Optional.empty());
Review Comment:
why here is empty, time filter cannot push down?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/tsfile/TsFileTableSizeCacheReader.java:
##########
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile;
+
+import org.apache.iotdb.commons.exception.IoTDBRuntimeException;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.utils.MmapUtil;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheReader {
Review Comment:
```suggestion
public class TsFileTableSizeIndexReader {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/optimization/LimitOffsetPushDown.java:
##########
@@ -174,6 +180,23 @@ public PlanNode visitMultiChildProcess(MultiChildProcessNode node, RewriterConte
return node;
}
+ @Override
+ public PlanNode visitCollect(CollectNode node, RewriterContext context) {
+ PlanNode newNode = node.clone();
+ RewriterContext subContext = new RewriterContext(context.getAnalysis());
+ if (context.getLimit() > 0) {
+ subContext.setLimit(context.getLimit() + context.getOffset());
+ }
+ for (PlanNode child : node.getChildren()) {
+ newNode.addChild(child.accept(this, subContext));
+ }
+ if (node.getChildren().size() > 1) {
+ // keep parent limit/offset node
+ context.setEnablePushDown(false);
+ }
Review Comment:
when will collectnode has only one child?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/distribution/SourceRewriter.java:
##########
@@ -141,6 +142,18 @@ public List<PlanNode> visitMergeSort(MergeSortNode node, DistributionPlanContext
return Collections.singletonList(newRoot);
}
+ @Override
+ public List<PlanNode> visitCollect(CollectNode node, DistributionPlanContext context) {
+ CollectNode newRoot =
+ new CollectNode(
+ context.queryContext.getQueryId().genPlanNodeId(), node.getOutputColumnNames());
+ for (int i = 0; i < node.getChildren().size(); i++) {
+ List<PlanNode> rewroteNodes = rewrite(node.getChildren().get(i), context);
+ rewroteNodes.forEach(newRoot::addChild);
+ }
+ return Collections.singletonList(newRoot);
+ }
Review Comment:
Why need to override it? what's the difference from the BaseSourceRewriter.defaultRewrite()
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/object/IObjectTableSizeCacheReader.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object;
+
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+
+import java.io.IOException;
+
+public interface IObjectTableSizeCacheReader extends AutoCloseable {
+ boolean loadObjectFileTableSize(
Review Comment:
add java doc for this method
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
+ }
+
+ public void remove(String database, int regionId) {
+ RemoveRegionOperation operation = new RemoveRegionOperation(database, regionId);
+ if (!addOperationToQueue(operation)) {
+ return;
+ }
+ try {
+ operation.future.get(5, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when remove TableDiskUsageCache.", e);
+ }
+ }
+
+ protected boolean addOperationToQueue(Operation operation) {
+ if (failedToRecover || stop) {
+ return false;
+ }
+ try {
+ queue.put(operation);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ return true;
+ }
+
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ public void close() {
+ if (scheduledExecutorService == null) {
+ return;
+ }
+ try {
+ stop = true;
+ scheduledExecutorService.shutdown();
+ scheduledExecutorService.awaitTermination(5, TimeUnit.SECONDS);
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ writerMap.clear();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ @TestOnly
+ public void ensureRunning() {
+ stop = false;
+ failedToRecover = false;
+ if (scheduledExecutorService.isTerminated()) {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+ }
+
+ protected DataRegionTableSizeCacheWriter createWriter(
+ String database, int regionId, DataRegion region) {
+ return new DataRegionTableSizeCacheWriter(database, regionId, region);
+ }
+
+ protected TsFileTableSizeCacheReader createTsFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ TsFileTableDiskUsageCacheWriter tsFileCacheWriter = dataRegionWriter.tsFileCacheWriter;
+ return new TsFileTableSizeCacheReader(
+ tsFileCacheWriter.keyFileLength(),
+ tsFileCacheWriter.getKeyFile(),
+ tsFileCacheWriter.valueFileLength(),
+ tsFileCacheWriter.getValueFile(),
+ regionId);
+ }
+
+ protected IObjectTableSizeCacheReader createObjectFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ return new EmptyObjectTableSizeCacheReader();
+ }
+
+ protected abstract static class Operation {
+ protected final String database;
+ protected final int regionId;
+
+ protected Operation(String database, int regionId) {
+ this.database = database;
+ this.regionId = regionId;
+ }
+
+ public int getRegionId() {
+ return regionId;
+ }
+
+ public String getDatabase() {
+ return database;
+ }
+
+ public abstract void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException;
+ }
+
+ protected static class StartReadOperation extends Operation {
+ protected final DataRegion region;
+ protected final boolean readTsFileCache;
+ protected final boolean readObjectFileCache;
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> future =
+ new CompletableFuture<>();
+
+ public StartReadOperation(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ this.readTsFileCache = readTsFileCache;
+ this.readObjectFileCache = readObjectFileCache;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter writer = tableDiskUsageCache.writerMap.get(regionId);
+ try {
+ if (writer == null || writer.getRemovedFuture() != null) {
+ // region is removing or removed
+ future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, regionId),
+ new EmptyObjectTableSizeCacheReader()));
+ return;
+ }
+ writer.increaseActiveReaderNum();
+ // Flush buffered writes to ensure readers observe a consistent snapshot
+ writer.flush();
+ TsFileTableSizeCacheReader tsFileTableSizeCacheReader =
+ readTsFileCache ? tableDiskUsageCache.createTsFileCacheReader(writer, regionId) : null;
+ IObjectTableSizeCacheReader objectTableSizeCacheReader =
+ readObjectFileCache
+ ? tableDiskUsageCache.createObjectFileCacheReader(writer, regionId)
+ : null;
+ future.complete(new Pair<>(tsFileTableSizeCacheReader, objectTableSizeCacheReader));
+ } catch (Throwable t) {
+ future.completeExceptionally(t);
+ }
+ }
+ }
+
+ private static class EndReadOperation extends Operation {
+ protected final DataRegion region;
+
+ public EndReadOperation(DataRegion dataRegion) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ tableDiskUsageCache.writerMap.computeIfPresent(
+ regionId,
+ (k, writer) -> {
+ if (writer.dataRegion != region) {
+ return writer;
+ }
+ writer.decreaseActiveReaderNum();
+ // Complete pending remove when the last reader exits
+ if (writer.getRemovedFuture() != null) {
+ writer.close();
+ writer.getRemovedFuture().complete(null);
+ writer.setRemovedFuture(null);
+ return null;
+ }
+ return writer;
+ });
+ }
+ }
+
+ private static class WriteOperation extends Operation {
+
+ private final TsFileID tsFileID;
+ private final Map<String, Long> tableSizeMap;
+
+ protected WriteOperation(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ super(database, tsFileID.regionId);
+ this.tsFileID = tsFileID;
+ this.tableSizeMap = tableSizeMap;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter dataRegionTableSizeCacheWriter =
+ tableDiskUsageCache.writerMap.get(regionId);
+ if (dataRegionTableSizeCacheWriter != null) {
+ dataRegionTableSizeCacheWriter.tsFileCacheWriter.write(tsFileID, tableSizeMap);
+ }
+ }
+ }
+
+ private static class ReplaceTsFileOperation extends Operation {
+ private final TsFileID originTsFileID;
+ private final TsFileID newTsFileID;
+
+ public ReplaceTsFileOperation(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ super(database, originTsFileID.regionId);
+ this.originTsFileID = originTsFileID;
+ this.newTsFileID = newTsFileID;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter writer = tableDiskUsageCache.writerMap.get(regionId);
+ if (writer != null) {
+ writer.tsFileCacheWriter.write(originTsFileID, newTsFileID);
+ }
+ }
+ }
+
+ protected static class RegisterRegionOperation extends Operation {
+
+ protected final DataRegion dataRegion;
+ protected final CompletableFuture<Void> future = new CompletableFuture<>();
+
+ public RegisterRegionOperation(DataRegion dataRegion) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.dataRegion = dataRegion;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) {
+ tableDiskUsageCache.writerMap.computeIfAbsent(
+ regionId, regionId -> tableDiskUsageCache.createWriter(database, regionId, dataRegion));
+ future.complete(null);
+ }
+
+ public CompletableFuture<Void> getFuture() {
+ return future;
+ }
+ }
+
+ private static class RemoveRegionOperation extends Operation {
+
+ private final CompletableFuture<Void> future = new CompletableFuture<>();
+
+ private RemoveRegionOperation(String database, int regionId) {
+ super(database, regionId);
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) {
+ tableDiskUsageCache.writerMap.computeIfPresent(
+ regionId,
+ (k, writer) -> {
+ if (writer.getActiveReaderNum() > 0) {
+ // If there are active readers, defer removal until all readers finish
+ writer.setRemovedFuture(future);
+ return writer;
+ }
+ writer.close();
+ future.complete(null);
+ return null;
+ });
+ }
+ }
+
+ public static TableDiskUsageCache getInstance() {
+ return TableDiskUsageCache.InstanceHolder.INSTANCE;
+ }
+
+ private static class InstanceHolder {
+ private InstanceHolder() {}
+
+ private static final TableDiskUsageCache INSTANCE = loadInstance();
+
+ private static TableDiskUsageCache loadInstance() {
+ ServiceLoader<TableDiskUsageCacheProvider> loader =
+ ServiceLoader.load(TableDiskUsageCacheProvider.class);
+ for (TableDiskUsageCacheProvider provider : loader) {
+ return provider.create();
+ }
+ return new DefaultTableDiskUsageCacheProvider().create();
+ }
+ }
+
+ protected static class DataRegionTableSizeCacheWriter {
+ protected final DataRegion dataRegion;
+ protected final TsFileTableDiskUsageCacheWriter tsFileCacheWriter;
+ protected int activeReaderNum = 0;
+ protected CompletableFuture<Void> removedFuture;
+
+ protected DataRegionTableSizeCacheWriter(String database, int regionId, DataRegion dataRegion) {
+ this.dataRegion = dataRegion;
+ this.tsFileCacheWriter = new TsFileTableDiskUsageCacheWriter(database, regionId);
+ }
+
+ public void increaseActiveReaderNum() {
+ activeReaderNum++;
+ }
+
+ public void decreaseActiveReaderNum() {
+ if (activeReaderNum > 0) {
+ activeReaderNum--;
+ }
Review Comment:
else should at least print a error log, indicate that there may be bug
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
Review Comment:
what if flush happens just every 1s? I think we need another condition to trigger snapshot:
record cpu time since last period, like since last PeriodicCheck, time flies 10s, but only 1s(10%) used by operation apply, then I think we can do snapshot.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
+ }
+
+ public void remove(String database, int regionId) {
+ RemoveRegionOperation operation = new RemoveRegionOperation(database, regionId);
+ if (!addOperationToQueue(operation)) {
+ return;
+ }
+ try {
+ operation.future.get(5, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when remove TableDiskUsageCache.", e);
+ }
+ }
+
+ protected boolean addOperationToQueue(Operation operation) {
+ if (failedToRecover || stop) {
+ return false;
+ }
+ try {
+ queue.put(operation);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ return true;
+ }
+
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ public void close() {
+ if (scheduledExecutorService == null) {
+ return;
+ }
+ try {
+ stop = true;
+ scheduledExecutorService.shutdown();
+ scheduledExecutorService.awaitTermination(5, TimeUnit.SECONDS);
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ writerMap.clear();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ @TestOnly
+ public void ensureRunning() {
+ stop = false;
+ failedToRecover = false;
+ if (scheduledExecutorService.isTerminated()) {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+ }
+
+ protected DataRegionTableSizeCacheWriter createWriter(
+ String database, int regionId, DataRegion region) {
+ return new DataRegionTableSizeCacheWriter(database, regionId, region);
+ }
+
+ protected TsFileTableSizeCacheReader createTsFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ TsFileTableDiskUsageCacheWriter tsFileCacheWriter = dataRegionWriter.tsFileCacheWriter;
+ return new TsFileTableSizeCacheReader(
+ tsFileCacheWriter.keyFileLength(),
+ tsFileCacheWriter.getKeyFile(),
+ tsFileCacheWriter.valueFileLength(),
+ tsFileCacheWriter.getValueFile(),
+ regionId);
+ }
+
+ protected IObjectTableSizeCacheReader createObjectFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ return new EmptyObjectTableSizeCacheReader();
+ }
+
+ protected abstract static class Operation {
+ protected final String database;
+ protected final int regionId;
+
+ protected Operation(String database, int regionId) {
+ this.database = database;
+ this.regionId = regionId;
+ }
+
+ public int getRegionId() {
+ return regionId;
+ }
+
+ public String getDatabase() {
+ return database;
+ }
+
+ public abstract void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException;
+ }
+
+ protected static class StartReadOperation extends Operation {
+ protected final DataRegion region;
+ protected final boolean readTsFileCache;
+ protected final boolean readObjectFileCache;
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> future =
+ new CompletableFuture<>();
+
+ public StartReadOperation(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ this.readTsFileCache = readTsFileCache;
+ this.readObjectFileCache = readObjectFileCache;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter writer = tableDiskUsageCache.writerMap.get(regionId);
+ try {
+ if (writer == null || writer.getRemovedFuture() != null) {
+ // region is removing or removed
+ future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, regionId),
+ new EmptyObjectTableSizeCacheReader()));
+ return;
+ }
+ writer.increaseActiveReaderNum();
+ // Flush buffered writes to ensure readers observe a consistent snapshot
+ writer.flush();
+ TsFileTableSizeCacheReader tsFileTableSizeCacheReader =
+ readTsFileCache ? tableDiskUsageCache.createTsFileCacheReader(writer, regionId) : null;
+ IObjectTableSizeCacheReader objectTableSizeCacheReader =
+ readObjectFileCache
+ ? tableDiskUsageCache.createObjectFileCacheReader(writer, regionId)
+ : null;
+ future.complete(new Pair<>(tsFileTableSizeCacheReader, objectTableSizeCacheReader));
+ } catch (Throwable t) {
+ future.completeExceptionally(t);
+ }
+ }
+ }
+
+ private static class EndReadOperation extends Operation {
+ protected final DataRegion region;
+
+ public EndReadOperation(DataRegion dataRegion) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ tableDiskUsageCache.writerMap.computeIfPresent(
+ regionId,
+ (k, writer) -> {
+ if (writer.dataRegion != region) {
+ return writer;
+ }
+ writer.decreaseActiveReaderNum();
+ // Complete pending remove when the last reader exits
+ if (writer.getRemovedFuture() != null) {
Review Comment:
only the last active reader should do it?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/DefaultTableDiskUsageCacheProvider.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+public class DefaultTableDiskUsageCacheProvider implements TableDiskUsageCacheProvider {
Review Comment:
```suggestion
public class DefaultTableDiskUsageIndexProvider implements TableDiskUsageCacheProvider {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/tsfile/TsFileTableDiskUsageCacheWriter.java:
##########
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.utils.FileUtils;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+
+import org.apache.tsfile.utils.Pair;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Stream;
+
+public class TsFileTableDiskUsageCacheWriter extends AbstractTableSizeCacheWriter {
Review Comment:
```suggestion
public class TsFileTableDiskUsageIndexWriter extends AbstractTableSizeCacheWriter {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/InformationSchemaContentSupplierFactory.java:
##########
@@ -1165,7 +1215,297 @@ public boolean hasNext() {
}
}
- private abstract static class TsBlockSupplier implements Iterator<TsBlock> {
+ private static class TableDiskUsageSupplier implements IInformationSchemaContentSupplier {
+ private final List<TSDataType> dataTypes;
+ private final Map<String, List<TTableInfo>> databaseTableInfoMap;
+ private final Filter pushDownFilter;
+ private final PaginationController paginationController;
+ private final OperatorContext operatorContext;
+
+ private DataRegion currentDataRegion;
+ private boolean currentDatabaseOnlyHasOneTable;
+
+ private TableDiskUsageCacheReader currentDataRegionCacheReader;
+ private DataRegionTableSizeQueryContext currentDataRegionTableSizeQueryContext;
+
+ private final StorageEngineTimePartitionIterator dataRegionIterator;
+
+ private long prepareCacheReaderCostInNS = 0;
+ private long loadObjectFileCostInNS = 0;
+ private long prepareCachedTsFileIDCostInNS = 0;
+ private long checkAllFilesInTsFileManagerCostInNS = 0;
+ private long readTsFileCacheValueFilesCostInNS = 0;
+
+ private TableDiskUsageSupplier(
+ final List<TSDataType> dataTypes,
+ final UserEntity userEntity,
+ final Filter pushDownFilter,
+ final PaginationController paginationController,
+ final OperatorContext operatorContext,
+ final List<Integer> regionsForCurrentSubTask)
+ throws TException, ClientManagerException {
+ this.dataTypes = dataTypes;
+ this.pushDownFilter = pushDownFilter;
+ this.paginationController = paginationController;
+ this.operatorContext = operatorContext;
+ AuthorityChecker.getAccessControl().checkUserGlobalSysPrivilege(userEntity);
+ this.databaseTableInfoMap =
+ operatorContext.getInstanceContext().getDataNodeQueryContext().getDatabaseTableInfoMap();
+ Set<Integer> regions = new HashSet<>(regionsForCurrentSubTask);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.REGIONS_OF_CURRENT_SUB_TASK, regionsForCurrentSubTask.toString());
+ this.dataRegionIterator =
+ new StorageEngineTimePartitionIterator(
+ Optional.of(
+ dataRegion -> {
+ List<TTableInfo> tTableInfos =
+ databaseTableInfoMap.get(dataRegion.getDatabaseName());
+ if (tTableInfos == null || tTableInfos.isEmpty()) {
+ return false;
+ }
+ if (!regions.contains(dataRegion.getDataRegionId())) {
+ return false;
+ }
+ currentDataRegionTableSizeQueryContext =
+ new DataRegionTableSizeQueryContext(
+ false, operatorContext.getInstanceContext());
+ return true;
+ }),
+ Optional.empty());
+ }
+
+ @Override
+ public boolean hasNext() {
+ boolean result = hasNextInternal();
+ if (!result) {
+ updateSpecifiedInfo();
+ }
+ return result;
+ }
+
+ private boolean hasNextInternal() {
+ if (currentDataRegionCacheReader != null) {
+ return true;
+ }
+ if (!paginationController.hasCurLimit()) {
+ return false;
+ }
+ try {
+ while (dataRegionIterator.nextDataRegion()) {
+ currentDataRegion = dataRegionIterator.currentDataRegion();
+ for (Long timePartition : currentDataRegion.getTsFileManager().getTimePartitions()) {
+ Map<String, Long> tablesToScan = getTablesToScan(currentDataRegion, timePartition);
+ if (!tablesToScan.isEmpty()) {
+ currentDataRegionTableSizeQueryContext.addTimePartition(
+ timePartition, new TimePartitionTableSizeQueryContext(tablesToScan));
+ }
+ }
+ if (currentDataRegionTableSizeQueryContext.isEmpty()) {
+ continue;
+ }
+ currentDataRegionCacheReader =
+ new TableDiskUsageCacheReader(
+ currentDataRegion,
+ currentDataRegionTableSizeQueryContext,
+ currentDatabaseOnlyHasOneTable);
+ return true;
+ }
+ } catch (Exception e) {
+ closeDataRegionReader();
+ throw new IoTDBRuntimeException(
+ e.getMessage(), e, TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode());
+ }
+ return false;
+ }
+
+ private void updateSpecifiedInfo() {
+ if (operatorContext
+ .getSpecifiedInfo()
+ .containsKey(PlanGraphPrinter.PREPARE_CACHE_READER_COST)) {
+ return;
+ }
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.PREPARE_CACHE_READER_COST,
+ TimeUnit.NANOSECONDS.toMillis(prepareCacheReaderCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.LOAD_OBJECT_FILE_COST,
+ TimeUnit.NANOSECONDS.toMillis(loadObjectFileCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.PREPARE_CACHED_TSFILE_ID_COST,
+ TimeUnit.NANOSECONDS.toMillis(prepareCachedTsFileIDCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.CHECK_ALL_FILES_IN_TSFILE_MANAGER_COST,
+ TimeUnit.NANOSECONDS.toMillis(checkAllFilesInTsFileManagerCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.READ_TSFILE_CACHE_VALUE_FILES_COST,
+ TimeUnit.NANOSECONDS.toMillis(readTsFileCacheValueFilesCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ }
+
+ private Map<String, Long> getTablesToScan(DataRegion dataRegion, long timePartition) {
+ String databaseName = dataRegion.getDatabaseName();
+ List<TTableInfo> tTableInfos = databaseTableInfoMap.get(databaseName);
+ if (tTableInfos == null || tTableInfos.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ Map<String, Long> tablesToScan = new TreeMap<>();
+ int totalValidTableCount = 0;
+ for (TTableInfo tTableInfo : tTableInfos) {
+ if (tTableInfo.getType() != TableType.BASE_TABLE.ordinal()) {
+ continue;
+ }
+ totalValidTableCount++;
+ if (pushDownFilter != null) {
+ Object[] row = new Object[5];
+ row[0] = new Binary(dataRegion.getDatabaseName(), TSFileConfig.STRING_CHARSET);
+ row[1] = new Binary(tTableInfo.getTableName(), TSFileConfig.STRING_CHARSET);
+ row[2] = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ row[3] = dataRegion.getDataRegionId();
+ row[4] = timePartition;
+ if (!pushDownFilter.satisfyRow(0, row)) {
+ continue;
+ }
+ }
+ if (!paginationController.hasCurLimit()) {
+ break;
+ }
+ if (paginationController.hasCurOffset()) {
+ paginationController.consumeOffset();
+ continue;
+ }
+ paginationController.consumeLimit();
+ tablesToScan.put(tTableInfo.getTableName(), 0L);
+ }
+ currentDatabaseOnlyHasOneTable = totalValidTableCount == 1;
+ return tablesToScan;
+ }
+
+ @Override
+ public TsBlock next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+
+ long maxRuntime = OperatorContext.getMaxRunTime().roundTo(TimeUnit.NANOSECONDS);
+ long start = System.nanoTime();
+ long prevStageEndTime = start;
+
+ try {
+ try {
+ if (!currentDataRegionCacheReader.prepareCacheReader(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ prepareCacheReaderCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.loadObjectFileTableSizeCache(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ loadObjectFileCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.prepareCachedTsFileIDKeys(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ prepareCachedTsFileIDCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.checkAllFilesInTsFileManager(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ checkAllFilesInTsFileManagerCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.readCacheValueFilesAndUpdateResultMap(
+ start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ readTsFileCacheValueFilesCostInNS += System.nanoTime() - prevStageEndTime;
+ }
+
+ return buildTsBlock();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return null;
+ } catch (Exception e) {
+ throw new IoTDBRuntimeException(
+ e.getMessage(), e, TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode());
+ }
+ }
+
+ private TsBlock buildTsBlock() {
+ TsBlockBuilder builder = new TsBlockBuilder(dataTypes);
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ currentDataRegionTableSizeQueryContext
+ .getTimePartitionTableSizeQueryContextMap()
+ .entrySet()) {
+ long timePartition = entry.getKey();
+ for (Map.Entry<String, Long> tableSizeEntry :
+ entry.getValue().getTableSizeResultMap().entrySet()) {
+ String tableName = tableSizeEntry.getKey();
+ long size = tableSizeEntry.getValue();
+ builder.getTimeColumnBuilder().writeLong(0);
+ ColumnBuilder[] columns = builder.getValueColumnBuilders();
+
+ columns[0].writeBinary(
+ new Binary(currentDataRegion.getDatabaseName(), TSFileConfig.STRING_CHARSET));
+ columns[1].writeBinary(new Binary(tableName, TSFileConfig.STRING_CHARSET));
+ columns[2].writeInt(IoTDBDescriptor.getInstance().getConfig().getDataNodeId());
+ columns[3].writeInt(currentDataRegion.getDataRegionId());
+ columns[4].writeLong(timePartition);
+ columns[5].writeLong(size);
+ builder.declarePosition();
+ }
+ }
+ closeDataRegionReader();
+ return builder.build();
+ }
+
+ @Override
+ public void close() throws IOException {
+ closeDataRegionReader();
+ }
+
+ private void closeDataRegionReader() {
+ if (currentDataRegionCacheReader == null) {
+ return;
+ }
+ try {
+ currentDataRegionCacheReader.close();
+ currentDataRegionCacheReader = null;
+ } catch (IOException ignored) {
Review Comment:
why ignore? should at least logger.info or warn
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
Review Comment:
what if we rewrite data to some inactive region after a long time?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/AbstractTableSizeCacheWriter.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.db.storageengine.StorageEngine;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+public abstract class AbstractTableSizeCacheWriter {
Review Comment:
```suggestion
public abstract class AbstractTableSizeIndexWriter {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCacheReader.java:
##########
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.TableDiskUsageStatisticUtil;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+
+public class TableDiskUsageCacheReader implements Closeable {
+
+ private final DataRegion dataRegion;
+ private final DataRegionTableSizeQueryContext dataRegionContext;
+
+ private CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>>
+ prepareReaderFuture;
+ private TsFileTableSizeCacheReader tsFileTableSizeCacheReader;
+ private IObjectTableSizeCacheReader objectTableSizeCacheReader;
+
+ private boolean objectFileSizeLoaded = false;
+ private boolean tsFileIdKeysPrepared = false;
+ private boolean allTsFileResourceChecked = false;
+
+ private final Iterator<Map.Entry<Long, TimePartitionTableSizeQueryContext>> timePartitionIterator;
+
+ private final boolean currentDatabaseOnlyHasOneTable;
+ private TableDiskUsageStatisticUtil tableDiskUsageStatisticUtil;
+
+ private final List<Pair<TsFileID, Long>> tsFilesToQueryInCache = new ArrayList<>();
+ private Iterator<Pair<TsFileID, Long>> tsFilesToQueryInCacheIterator = null;
+
+ public TableDiskUsageCacheReader(
+ DataRegion dataRegion,
+ DataRegionTableSizeQueryContext dataRegionContext,
+ boolean databaseHasOnlyOneTable) {
+ this.dataRegion = dataRegion;
+ this.dataRegionContext = dataRegionContext;
+ this.currentDatabaseOnlyHasOneTable = databaseHasOnlyOneTable;
+ if (dataRegionContext.isNeedAllData()) {
+ dataRegionContext.addAllTimePartitionsInTsFileManager(dataRegion.getTsFileManager());
+ }
+ this.timePartitionIterator =
+ dataRegionContext.getTimePartitionTableSizeQueryContextMap().entrySet().iterator();
+ dataRegionContext.reserveMemoryForResultMap();
+ }
+
+ public boolean prepareCacheReader(long startTime, long maxRunTime) throws Exception {
+ if (this.tsFileTableSizeCacheReader == null) {
+ this.prepareReaderFuture =
+ this.prepareReaderFuture == null
+ ? TableDiskUsageCache.getInstance().startRead(dataRegion, true, true)
+ : this.prepareReaderFuture;
+ do {
+ if (prepareReaderFuture.isDone()) {
+ Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader> readerPair =
+ prepareReaderFuture.get();
+ this.tsFileTableSizeCacheReader = readerPair.left;
+ this.tsFileTableSizeCacheReader.openKeyFile();
+ this.objectTableSizeCacheReader = readerPair.right;
+ break;
+ } else {
+ Thread.sleep(1);
Review Comment:
```suggestion
prepareReaderFuture.wait(System.nanoTime() - startTime);
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/writer/AbstractCrossCompactionWriter.java:
##########
@@ -171,6 +171,9 @@ public void endFile() throws IOException {
// set empty target file to DELETED
if (isEmptyFile[i]) {
targetResources.get(i).forceMarkDeleted();
+ } else if (compactionTaskSummary != null) {
+ compactionTaskSummary.recordTargetTsFileTableSizeMap(
+ targetResources.get(i), targetFileWriters.get(i).getTableSizeMap());
Review Comment:
else if or standalone if branch? if targetResources.get(i) is empty, should we recordTargetTsFileTableSizeMap for this compaction?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/object/IObjectTableSizeCacheReader.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object;
+
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+
+import java.io.IOException;
+
+public interface IObjectTableSizeCacheReader extends AutoCloseable {
+ boolean loadObjectFileTableSize(
+ DataRegionTableSizeQueryContext dataRegionContext, long startTime, long maxRunTime)
+ throws IOException;
+
+ @Override
+ void close();
Review Comment:
why you need to override it in an interface?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
Review Comment:
```suggestion
public class TableDiskUsageIndex {
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -3652,29 +3653,82 @@ public Analysis visitShowQueries(
analysis.setSourceExpressions(sourceExpressions);
sourceExpressions.forEach(expression -> analyzeExpressionType(analysis, expression));
- analyzeWhere(analysis, showQueriesStatement);
+ if (!analyzeWhere(
+ analysis,
+ showQueriesStatement.getWhereCondition(),
+ ColumnHeaderConstant.showQueriesColumnHeaders)) {
+ showQueriesStatement.setWhereCondition(null);
+ }
analysis.setMergeOrderParameter(new OrderByParameter(showQueriesStatement.getSortItemList()));
return analysis;
}
- private void analyzeWhere(Analysis analysis, ShowQueriesStatement showQueriesStatement) {
- WhereCondition whereCondition = showQueriesStatement.getWhereCondition();
+ @Override
+ public Analysis visitShowDiskUsage(
+ ShowDiskUsageStatement showDiskUsageStatement, MPPQueryContext context) {
+ Analysis analysis = new Analysis();
+ analysis.setRealStatement(showDiskUsageStatement);
+ analysis.setRespDatasetHeader(DatasetHeaderFactory.getShowDiskUsageHeader());
+ analysis.setVirtualSource(true);
+
+ List<TDataNodeLocation> allReadableDataNodeLocations = getReadableDataNodeLocations();
+ if (allReadableDataNodeLocations.isEmpty()) {
+ throw new StatementAnalyzeException("no Running DataNodes");
+ }
+ analysis.setReadableDataNodeLocations(allReadableDataNodeLocations);
+
+ Set<Expression> sourceExpressions = new HashSet<>();
+ for (ColumnHeader columnHeader : analysis.getRespDatasetHeader().getColumnHeaders()) {
+ sourceExpressions.add(
+ TimeSeriesOperand.constructColumnHeaderExpression(
+ columnHeader.getColumnName(), columnHeader.getColumnType()));
+ }
+ analysis.setSourceExpressions(sourceExpressions);
+ sourceExpressions.forEach(expression -> analyzeExpressionType(analysis, expression));
+
+ if (!analyzeWhere(
+ analysis,
+ showDiskUsageStatement.getWhereCondition(),
+ ColumnHeaderConstant.showDiskUsageColumnHeaders)) {
+ showDiskUsageStatement.setWhereCondition(null);
+ }
+
+ analysis.setMergeOrderParameter(new OrderByParameter(showDiskUsageStatement.getSortItemList()));
+
+ return analysis;
+ }
+
+ private boolean analyzeWhere(
Review Comment:
add java doc about the method, at least including when return true, when retrun false
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
Review Comment:
why ignore the return value of `addOperationToQueue`? what if it's false? and the future in `RegisterRegionOperation` is never used.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
Review Comment:
```suggestion
ThreadName.TABLE_SIZE_INDEX_RECORD.getName());
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
+ }
+
+ public void remove(String database, int regionId) {
+ RemoveRegionOperation operation = new RemoveRegionOperation(database, regionId);
+ if (!addOperationToQueue(operation)) {
+ return;
+ }
+ try {
+ operation.future.get(5, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when remove TableDiskUsageCache.", e);
+ }
+ }
+
+ protected boolean addOperationToQueue(Operation operation) {
+ if (failedToRecover || stop) {
+ return false;
+ }
+ try {
+ queue.put(operation);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ return true;
+ }
+
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ public void close() {
+ if (scheduledExecutorService == null) {
+ return;
+ }
+ try {
+ stop = true;
+ scheduledExecutorService.shutdown();
+ scheduledExecutorService.awaitTermination(5, TimeUnit.SECONDS);
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ writerMap.clear();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ @TestOnly
+ public void ensureRunning() {
+ stop = false;
+ failedToRecover = false;
+ if (scheduledExecutorService.isTerminated()) {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+ }
+
+ protected DataRegionTableSizeCacheWriter createWriter(
+ String database, int regionId, DataRegion region) {
+ return new DataRegionTableSizeCacheWriter(database, regionId, region);
+ }
+
+ protected TsFileTableSizeCacheReader createTsFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ TsFileTableDiskUsageCacheWriter tsFileCacheWriter = dataRegionWriter.tsFileCacheWriter;
+ return new TsFileTableSizeCacheReader(
+ tsFileCacheWriter.keyFileLength(),
+ tsFileCacheWriter.getKeyFile(),
+ tsFileCacheWriter.valueFileLength(),
+ tsFileCacheWriter.getValueFile(),
+ regionId);
+ }
+
+ protected IObjectTableSizeCacheReader createObjectFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ return new EmptyObjectTableSizeCacheReader();
+ }
+
+ protected abstract static class Operation {
+ protected final String database;
+ protected final int regionId;
+
+ protected Operation(String database, int regionId) {
+ this.database = database;
+ this.regionId = regionId;
+ }
+
+ public int getRegionId() {
+ return regionId;
+ }
+
+ public String getDatabase() {
+ return database;
+ }
+
+ public abstract void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException;
+ }
+
+ protected static class StartReadOperation extends Operation {
+ protected final DataRegion region;
+ protected final boolean readTsFileCache;
+ protected final boolean readObjectFileCache;
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> future =
Review Comment:
```suggestion
private final CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> future =
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
+ }
+
+ public void remove(String database, int regionId) {
+ RemoveRegionOperation operation = new RemoveRegionOperation(database, regionId);
+ if (!addOperationToQueue(operation)) {
+ return;
+ }
+ try {
+ operation.future.get(5, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when remove TableDiskUsageCache.", e);
+ }
+ }
+
+ protected boolean addOperationToQueue(Operation operation) {
+ if (failedToRecover || stop) {
+ return false;
+ }
+ try {
+ queue.put(operation);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ return true;
+ }
+
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ public void close() {
+ if (scheduledExecutorService == null) {
+ return;
+ }
+ try {
+ stop = true;
+ scheduledExecutorService.shutdown();
+ scheduledExecutorService.awaitTermination(5, TimeUnit.SECONDS);
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ writerMap.clear();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ @TestOnly
+ public void ensureRunning() {
+ stop = false;
+ failedToRecover = false;
+ if (scheduledExecutorService.isTerminated()) {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+ }
+
+ protected DataRegionTableSizeCacheWriter createWriter(
+ String database, int regionId, DataRegion region) {
+ return new DataRegionTableSizeCacheWriter(database, regionId, region);
+ }
+
+ protected TsFileTableSizeCacheReader createTsFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ TsFileTableDiskUsageCacheWriter tsFileCacheWriter = dataRegionWriter.tsFileCacheWriter;
+ return new TsFileTableSizeCacheReader(
+ tsFileCacheWriter.keyFileLength(),
+ tsFileCacheWriter.getKeyFile(),
+ tsFileCacheWriter.valueFileLength(),
+ tsFileCacheWriter.getValueFile(),
+ regionId);
+ }
+
+ protected IObjectTableSizeCacheReader createObjectFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ return new EmptyObjectTableSizeCacheReader();
+ }
+
+ protected abstract static class Operation {
+ protected final String database;
+ protected final int regionId;
+
+ protected Operation(String database, int regionId) {
+ this.database = database;
+ this.regionId = regionId;
+ }
+
+ public int getRegionId() {
+ return regionId;
+ }
+
+ public String getDatabase() {
+ return database;
+ }
+
+ public abstract void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException;
+ }
+
+ protected static class StartReadOperation extends Operation {
+ protected final DataRegion region;
+ protected final boolean readTsFileCache;
+ protected final boolean readObjectFileCache;
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> future =
+ new CompletableFuture<>();
+
+ public StartReadOperation(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ this.readTsFileCache = readTsFileCache;
+ this.readObjectFileCache = readObjectFileCache;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter writer = tableDiskUsageCache.writerMap.get(regionId);
+ try {
+ if (writer == null || writer.getRemovedFuture() != null) {
+ // region is removing or removed
+ future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, regionId),
+ new EmptyObjectTableSizeCacheReader()));
+ return;
+ }
+ writer.increaseActiveReaderNum();
+ // Flush buffered writes to ensure readers observe a consistent snapshot
+ writer.flush();
+ TsFileTableSizeCacheReader tsFileTableSizeCacheReader =
+ readTsFileCache ? tableDiskUsageCache.createTsFileCacheReader(writer, regionId) : null;
+ IObjectTableSizeCacheReader objectTableSizeCacheReader =
+ readObjectFileCache
+ ? tableDiskUsageCache.createObjectFileCacheReader(writer, regionId)
+ : null;
+ future.complete(new Pair<>(tsFileTableSizeCacheReader, objectTableSizeCacheReader));
+ } catch (Throwable t) {
+ future.completeExceptionally(t);
+ }
+ }
+ }
+
+ private static class EndReadOperation extends Operation {
+ protected final DataRegion region;
+
+ public EndReadOperation(DataRegion dataRegion) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ tableDiskUsageCache.writerMap.computeIfPresent(
+ regionId,
+ (k, writer) -> {
+ if (writer.dataRegion != region) {
+ return writer;
+ }
+ writer.decreaseActiveReaderNum();
+ // Complete pending remove when the last reader exits
+ if (writer.getRemovedFuture() != null) {
+ writer.close();
+ writer.getRemovedFuture().complete(null);
+ writer.setRemovedFuture(null);
+ return null;
+ }
+ return writer;
+ });
+ }
+ }
+
+ private static class WriteOperation extends Operation {
+
+ private final TsFileID tsFileID;
+ private final Map<String, Long> tableSizeMap;
+
+ protected WriteOperation(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ super(database, tsFileID.regionId);
+ this.tsFileID = tsFileID;
+ this.tableSizeMap = tableSizeMap;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter dataRegionTableSizeCacheWriter =
+ tableDiskUsageCache.writerMap.get(regionId);
+ if (dataRegionTableSizeCacheWriter != null) {
+ dataRegionTableSizeCacheWriter.tsFileCacheWriter.write(tsFileID, tableSizeMap);
+ }
Review Comment:
else should print warn or error log? It's a critical bug
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TableDiskUsageCache.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.commons.concurrent.IoTDBThreadPoolFactory;
+import org.apache.iotdb.commons.concurrent.ThreadName;
+import org.apache.iotdb.commons.utils.TestOnly;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.EmptyObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object.IObjectTableSizeCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.ServiceLoader;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+public class TableDiskUsageCache {
+ protected static final Logger LOGGER = LoggerFactory.getLogger(TableDiskUsageCache.class);
+ protected final BlockingQueue<Operation> queue = new LinkedBlockingQueue<>(1000);
+ // regionId -> writer mapping
+ protected final Map<Integer, DataRegionTableSizeCacheWriter> writerMap = new HashMap<>();
+ protected ScheduledExecutorService scheduledExecutorService;
+ private int processedOperationCountSinceLastPeriodicCheck = 0;
+ protected volatile boolean failedToRecover = false;
+ private volatile boolean stop = false;
+
+ protected TableDiskUsageCache() {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+
+ protected void run() {
+ try {
+ while (!stop) {
+ try {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ syncTsFileTableSizeCacheIfNecessary(writer);
+ persistPendingObjectDeltasIfNecessary(writer);
+ }
+ Operation operation = queue.poll(1, TimeUnit.SECONDS);
+ if (operation != null) {
+ operation.apply(this);
+ processedOperationCountSinceLastPeriodicCheck++;
+ }
+ if (operation == null || processedOperationCountSinceLastPeriodicCheck % 1000 == 0) {
+ performPeriodicMaintenance();
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return;
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when apply TableDiskUsageCache operation.", e);
+ }
+ }
+ } finally {
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ }
+ }
+
+ private void performPeriodicMaintenance() {
+ checkAndMayCloseIdleWriter();
+ compactIfNecessary(TimeUnit.SECONDS.toMillis(1));
+ processedOperationCountSinceLastPeriodicCheck = 0;
+ }
+
+ /**
+ * Any unrecoverable error in a single writer will mark the whole TableDiskUsageCache as failed
+ * and disable further operations.
+ */
+ protected void failedToRecover(Exception e) {
+ failedToRecover = true;
+ LOGGER.error("Failed to recover TableDiskUsageCache", e);
+ }
+
+ protected void syncTsFileTableSizeCacheIfNecessary(DataRegionTableSizeCacheWriter writer) {
+ try {
+ writer.tsFileCacheWriter.syncIfNecessary();
+ } catch (IOException e) {
+ LOGGER.warn("Failed to sync tsfile table size cache.", e);
+ }
+ }
+
+ // Hook for subclasses to persist pending object table size deltas. No-op by default.
+ protected void persistPendingObjectDeltasIfNecessary(DataRegionTableSizeCacheWriter writer) {}
+
+ protected void compactIfNecessary(long maxRunTime) {
+ if (!StorageEngine.getInstance().isReadyForReadAndWrite()) {
+ return;
+ }
+ long startTime = System.currentTimeMillis();
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ if (System.currentTimeMillis() - startTime > maxRunTime) {
+ break;
+ }
+ if (writer.getActiveReaderNum() > 0) {
+ continue;
+ }
+ writer.compactIfNecessary();
+ }
+ }
+
+ protected void checkAndMayCloseIdleWriter() {
+ for (DataRegionTableSizeCacheWriter writer : writerMap.values()) {
+ writer.closeIfIdle();
+ }
+ }
+
+ public void write(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ if (tableSizeMap == null || tableSizeMap.isEmpty()) {
+ // tree model
+ return;
+ }
+ addOperationToQueue(new WriteOperation(database, tsFileID, tableSizeMap));
+ }
+
+ public void write(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ addOperationToQueue(new ReplaceTsFileOperation(database, originTsFileID, newTsFileID));
+ }
+
+ public void writeObjectDelta(
+ String database, int regionId, long timePartition, String table, long size, int num) {
+ throw new UnsupportedOperationException("writeObjectDelta");
+ }
+
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> startRead(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ StartReadOperation operation =
+ new StartReadOperation(dataRegion, readTsFileCache, readObjectFileCache);
+ if (!addOperationToQueue(operation)) {
+ operation.future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, dataRegion.getDataRegionId()),
+ new EmptyObjectTableSizeCacheReader()));
+ }
+ return operation.future;
+ }
+
+ public void endRead(DataRegion dataRegion) {
+ EndReadOperation operation = new EndReadOperation(dataRegion);
+ addOperationToQueue(operation);
+ }
+
+ public void registerRegion(DataRegion region) {
+ RegisterRegionOperation operation = new RegisterRegionOperation(region);
+ if (!region.isTableModel()) {
+ return;
+ }
+ addOperationToQueue(operation);
+ }
+
+ public void remove(String database, int regionId) {
+ RemoveRegionOperation operation = new RemoveRegionOperation(database, regionId);
+ if (!addOperationToQueue(operation)) {
+ return;
+ }
+ try {
+ operation.future.get(5, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ } catch (Exception e) {
+ LOGGER.error("Meet exception when remove TableDiskUsageCache.", e);
+ }
+ }
+
+ protected boolean addOperationToQueue(Operation operation) {
+ if (failedToRecover || stop) {
+ return false;
+ }
+ try {
+ queue.put(operation);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return false;
+ }
+ return true;
+ }
+
+ public int getQueueSize() {
+ return queue.size();
+ }
+
+ public void close() {
+ if (scheduledExecutorService == null) {
+ return;
+ }
+ try {
+ stop = true;
+ scheduledExecutorService.shutdown();
+ scheduledExecutorService.awaitTermination(5, TimeUnit.SECONDS);
+ writerMap.values().forEach(DataRegionTableSizeCacheWriter::close);
+ writerMap.clear();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ }
+
+ @TestOnly
+ public void ensureRunning() {
+ stop = false;
+ failedToRecover = false;
+ if (scheduledExecutorService.isTerminated()) {
+ scheduledExecutorService =
+ IoTDBThreadPoolFactory.newSingleThreadScheduledExecutor(
+ ThreadName.FILE_TIME_INDEX_RECORD.getName());
+ scheduledExecutorService.submit(this::run);
+ }
+ }
+
+ protected DataRegionTableSizeCacheWriter createWriter(
+ String database, int regionId, DataRegion region) {
+ return new DataRegionTableSizeCacheWriter(database, regionId, region);
+ }
+
+ protected TsFileTableSizeCacheReader createTsFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ TsFileTableDiskUsageCacheWriter tsFileCacheWriter = dataRegionWriter.tsFileCacheWriter;
+ return new TsFileTableSizeCacheReader(
+ tsFileCacheWriter.keyFileLength(),
+ tsFileCacheWriter.getKeyFile(),
+ tsFileCacheWriter.valueFileLength(),
+ tsFileCacheWriter.getValueFile(),
+ regionId);
+ }
+
+ protected IObjectTableSizeCacheReader createObjectFileCacheReader(
+ DataRegionTableSizeCacheWriter dataRegionWriter, int regionId) {
+ return new EmptyObjectTableSizeCacheReader();
+ }
+
+ protected abstract static class Operation {
+ protected final String database;
+ protected final int regionId;
+
+ protected Operation(String database, int regionId) {
+ this.database = database;
+ this.regionId = regionId;
+ }
+
+ public int getRegionId() {
+ return regionId;
+ }
+
+ public String getDatabase() {
+ return database;
+ }
+
+ public abstract void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException;
+ }
+
+ protected static class StartReadOperation extends Operation {
+ protected final DataRegion region;
+ protected final boolean readTsFileCache;
+ protected final boolean readObjectFileCache;
+ public CompletableFuture<Pair<TsFileTableSizeCacheReader, IObjectTableSizeCacheReader>> future =
+ new CompletableFuture<>();
+
+ public StartReadOperation(
+ DataRegion dataRegion, boolean readTsFileCache, boolean readObjectFileCache) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ this.readTsFileCache = readTsFileCache;
+ this.readObjectFileCache = readObjectFileCache;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter writer = tableDiskUsageCache.writerMap.get(regionId);
+ try {
+ if (writer == null || writer.getRemovedFuture() != null) {
+ // region is removing or removed
+ future.complete(
+ new Pair<>(
+ new TsFileTableSizeCacheReader(0, null, 0, null, regionId),
+ new EmptyObjectTableSizeCacheReader()));
+ return;
+ }
+ writer.increaseActiveReaderNum();
+ // Flush buffered writes to ensure readers observe a consistent snapshot
+ writer.flush();
+ TsFileTableSizeCacheReader tsFileTableSizeCacheReader =
+ readTsFileCache ? tableDiskUsageCache.createTsFileCacheReader(writer, regionId) : null;
+ IObjectTableSizeCacheReader objectTableSizeCacheReader =
+ readObjectFileCache
+ ? tableDiskUsageCache.createObjectFileCacheReader(writer, regionId)
+ : null;
+ future.complete(new Pair<>(tsFileTableSizeCacheReader, objectTableSizeCacheReader));
+ } catch (Throwable t) {
+ future.completeExceptionally(t);
+ }
+ }
+ }
+
+ private static class EndReadOperation extends Operation {
+ protected final DataRegion region;
+
+ public EndReadOperation(DataRegion dataRegion) {
+ super(dataRegion.getDatabaseName(), dataRegion.getDataRegionId());
+ this.region = dataRegion;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ tableDiskUsageCache.writerMap.computeIfPresent(
+ regionId,
+ (k, writer) -> {
+ if (writer.dataRegion != region) {
+ return writer;
+ }
+ writer.decreaseActiveReaderNum();
+ // Complete pending remove when the last reader exits
+ if (writer.getRemovedFuture() != null) {
+ writer.close();
+ writer.getRemovedFuture().complete(null);
+ writer.setRemovedFuture(null);
+ return null;
+ }
+ return writer;
+ });
+ }
+ }
+
+ private static class WriteOperation extends Operation {
+
+ private final TsFileID tsFileID;
+ private final Map<String, Long> tableSizeMap;
+
+ protected WriteOperation(String database, TsFileID tsFileID, Map<String, Long> tableSizeMap) {
+ super(database, tsFileID.regionId);
+ this.tsFileID = tsFileID;
+ this.tableSizeMap = tableSizeMap;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter dataRegionTableSizeCacheWriter =
+ tableDiskUsageCache.writerMap.get(regionId);
+ if (dataRegionTableSizeCacheWriter != null) {
+ dataRegionTableSizeCacheWriter.tsFileCacheWriter.write(tsFileID, tableSizeMap);
+ }
+ }
+ }
+
+ private static class ReplaceTsFileOperation extends Operation {
+ private final TsFileID originTsFileID;
+ private final TsFileID newTsFileID;
+
+ public ReplaceTsFileOperation(String database, TsFileID originTsFileID, TsFileID newTsFileID) {
+ super(database, originTsFileID.regionId);
+ this.originTsFileID = originTsFileID;
+ this.newTsFileID = newTsFileID;
+ }
+
+ @Override
+ public void apply(TableDiskUsageCache tableDiskUsageCache) throws IOException {
+ DataRegionTableSizeCacheWriter writer = tableDiskUsageCache.writerMap.get(regionId);
+ if (writer != null) {
+ writer.tsFileCacheWriter.write(originTsFileID, newTsFileID);
+ }
Review Comment:
else should print warn or error log? It's a critical bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3857970206
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/17169?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `53.61702%` with `981 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 39.59%. Comparing base ([`e4ef4db`](https://app.codecov.io/gh/apache/iotdb/commit/e4ef4db9a692c6f25000e42eac86463335f82b66?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e7785f3`](https://app.codecov.io/gh/apache/iotdb/commit/e7785f35756fba80e7210e41e9722cf227e2b5e4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 21 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/17169?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ional/InformationSchemaContentSupplierFactory.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fsource%2Frelational%2FInformationSchemaContentSupplierFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3NvdXJjZS9yZWxhdGlvbmFsL0luZm9ybWF0aW9uU2NoZW1hQ29udGVudFN1cHBsaWVyRmFjdG9yeS5qYXZh) | 0.00% | [185 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ecution/operator/source/ShowDiskUsageOperator.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fsource%2FShowDiskUsageOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3NvdXJjZS9TaG93RGlza1VzYWdlT3BlcmF0b3IuamF2YQ==) | 0.00% | [77 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/dataregion/utils/TreeDiskUsageStatisticUtil.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Futils%2FTreeDiskUsageStatisticUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3V0aWxzL1RyZWVEaXNrVXNhZ2VTdGF0aXN0aWNVdGlsLmphdmE=) | 0.00% | [64 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...utils/tableDiskUsageCache/TableDiskUsageCache.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Futils%2FtableDiskUsageCache%2FTableDiskUsageCache.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3V0aWxzL3RhYmxlRGlza1VzYWdlQ2FjaGUvVGFibGVEaXNrVXNhZ2VDYWNoZS5qYXZh) | 76.59% | [55 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...nner/distribute/TableDistributedPlanGenerator.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fplanner%2Fdistribute%2FTableDistributedPlanGenerator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3BsYW5uZXIvZGlzdHJpYnV0ZS9UYWJsZURpc3RyaWJ1dGVkUGxhbkdlbmVyYXRvci5qYXZh) | 8.51% | [43 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...b/queryengine/plan/planner/LogicalPlanBuilder.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2FLogicalPlanBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL0xvZ2ljYWxQbGFuQnVpbGRlci5qYXZh) | 0.00% | [40 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ngine/dataregion/utils/DiskUsageStatisticUtil.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Futils%2FDiskUsageStatisticUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3V0aWxzL0Rpc2tVc2FnZVN0YXRpc3RpY1V0aWwuamF2YQ==) | 63.96% | [40 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...db/db/queryengine/plan/analyze/AnalyzeVisitor.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fanalyze%2FAnalyzeVisitor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9hbmFseXplL0FuYWx5emVWaXNpdG9yLmphdmE=) | 0.00% | [38 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...kUsageCache/tsfile/TsFileTableSizeCacheReader.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fstorageengine%2Fdataregion%2Futils%2FtableDiskUsageCache%2Ftsfile%2FTsFileTableSizeCacheReader.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvc3RvcmFnZWVuZ2luZS9kYXRhcmVnaW9uL3V0aWxzL3RhYmxlRGlza1VzYWdlQ2FjaGUvdHNmaWxlL1RzRmlsZVRhYmxlU2l6ZUNhY2hlUmVhZGVyLmphdmE=) | 81.12% | [37 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...an/planner/plan/node/source/ShowDiskUsageNode.java](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2Fplan%2Fnode%2Fsource%2FShowDiskUsageNode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL3BsYW4vbm9kZS9zb3VyY2UvU2hvd0Rpc2tVc2FnZU5vZGUuamF2YQ==) | 47.05% | [36 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [39 more](https://app.codecov.io/gh/apache/iotdb/pull/17169?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17169 +/- ##
============================================
+ Coverage 39.47% 39.59% +0.12%
Complexity 282 282
============================================
Files 5098 5120 +22
Lines 341273 343780 +2507
Branches 43471 43795 +324
============================================
+ Hits 134714 136120 +1406
- Misses 206559 207660 +1101
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/17169?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2772557905
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter5() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "2"
+ + TsFileTableDiskUsageCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "2"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
Review Comment:
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
Review Comment:
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
Review Comment:
Potential uncaught 'java.lang.NumberFormatException'.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
Review Comment:
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
Review Comment:
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
Review Comment:
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
Review Comment:
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
Review Comment:
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ @Override
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ @Override
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter5() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "2"
+ + TsFileTableDiskUsageCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "2"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ tempKeyFile2.createNewFile();
Review Comment:
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2938569499
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/object/IObjectTableSizeCacheReader.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object;
+
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+
+import java.io.IOException;
+
+public interface IObjectTableSizeCacheReader extends AutoCloseable {
+ boolean loadObjectFileTableSize(
+ DataRegionTableSizeQueryContext dataRegionContext, long startTime, long maxRunTime)
+ throws IOException;
+
+ @Override
+ void close();
Review Comment:
add comments about reasons
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3851798974
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3875240742
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-4066609872
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2767734010
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java:
##########
@@ -388,10 +414,31 @@ public class InformationSchema {
schemaTables.put(SERVICES, servicesTable);
}
+ static {
+ columnsThatSupportPushDownPredicate.put(
+ TABLE_DISK_USAGE,
+ ImmutableSet.of(
+ ColumnHeaderConstant.DATABASE,
+ ColumnHeaderConstant.TABLE_NAME_TABLE_MODEL,
+ ColumnHeaderConstant.DATA_NODE_ID_TABLE_MODEL,
+ ColumnHeaderConstant.REGION_ID_TABLE_MODEL,
+ ColumnHeaderConstant.TIME_PARTITION_TABLE_MODEL));
+
+ tablesThatSupportPushDownLimitOffset.add(TABLE_DISK_USAGE);
+ }
+
public static Map<String, TsTable> getSchemaTables() {
return schemaTables;
}
+ public static Set<String> getColumnsSupportPushDownPredicate(String tableName) {
+ return columnsThatSupportPushDownPredicate.getOrDefault(tableName, Collections.emptySet());
+ }
+
+ public static boolean supportsPushDownLimitOffset(String tableName) {
+ return columnsThatSupportPushDownPredicate.containsKey(tableName);
Review Comment:
The method `supportsPushDownLimitOffset` checks if a key exists in `columnsThatSupportPushDownPredicate` instead of checking `tablesThatSupportPushDownLimitOffset`. This will cause incorrect behavior for tables that support limit/offset push-down but are not in the predicate push-down map. The condition should be `tablesThatSupportPushDownLimitOffset.contains(tableName)`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/TimePartitionTableSizeQueryContext.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache;
+
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+
+import org.apache.tsfile.utils.Accountable;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class TimePartitionTableSizeQueryContext implements Accountable {
+ private static final long SHALLOW_SIZE =
+ RamUsageEstimator.shallowSizeOf(TimePartitionTableSizeQueryContext.class);
+
+ /**
+ * tableSizeResultMap serves as both: 1) result container 2) table filter when needAllData is
+ * false
+ */
+ private final Map<String, Long> tableSizeResultMap;
+
+ // tsFileIDOffsetInValueFileMap should be null at first
+ private Map<TsFileID, Long> tsFileIDOffsetInValueFileMap;
+
+ public TimePartitionTableSizeQueryContext(Map<String, Long> tableSizeResultMap) {
+ this.tableSizeResultMap = tableSizeResultMap;
+ }
+
+ public void addCachedTsFileIDAndOffsetInValueFile(TsFileID tsFileID, long offset) {
+ if (tsFileIDOffsetInValueFileMap == null) {
+ tsFileIDOffsetInValueFileMap = new HashMap<>();
+ }
+ tsFileIDOffsetInValueFileMap.put(tsFileID, offset);
+ }
+
+ public void replaceCachedTsFileID(TsFileID originTsFileID, TsFileID newTsFileID) {
+ if (tsFileIDOffsetInValueFileMap == null) {
+ return;
+ }
+ Long offset = tsFileIDOffsetInValueFileMap.remove(originTsFileID);
+ if (offset != null) {
+ tsFileIDOffsetInValueFileMap.put(newTsFileID, offset);
+ }
+ }
+
+ public void updateResult(String table, long size, boolean needAllData) {
+ if (needAllData) {
+ tableSizeResultMap.compute(table, (k, v) -> (v == null ? 0 : v) + size);
+ } else {
+ tableSizeResultMap.computeIfPresent(table, (k, v) -> v + size);
+ }
+ }
+
+ public Map<String, Long> getTableSizeResultMap() {
+ return tableSizeResultMap;
+ }
+
+ public Long getCachedTsFileIdOffset(TsFileID tsFileID) {
+ return tsFileIDOffsetInValueFileMap == null ? null : tsFileIDOffsetInValueFileMap.get(tsFileID);
+ }
+
+ public long getObjectFileSizeOfCurrentTimePartition() {
+ long size = 0;
+ for (Long value : tableSizeResultMap.values()) {
+ size += value;
+ }
+ return size;
+ }
+
+ @Override
+ public long ramBytesUsed() {
+ return SHALLOW_SIZE
+ + RamUsageEstimator.sizeOfMapWithKnownShallowSize(
+ tableSizeResultMap,
+ RamUsageEstimator.SHALLOW_SIZE_OF_HASHMAP,
+ RamUsageEstimator.SHALLOW_SIZE_OF_HASHMAP_ENTRY)
+ + ramBytesUsedOfTsFileIDOffsetMap();
+ }
+
+ // tsFileIDOffsetInValueFileMap should be null af first
Review Comment:
Typo in comment: "af" should be "at". The comment should read "tsFileIDOffsetInValueFileMap should be null at first".
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/DiskUsageStatisticUtil.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceContext;
+import org.apache.iotdb.db.storageengine.dataregion.read.control.FileReaderManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+
+import org.apache.tsfile.file.IMetadataIndexEntry;
+import org.apache.tsfile.file.header.ChunkGroupHeader;
+import org.apache.tsfile.file.metadata.IChunkMetadata;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.tsfile.read.TsFileSequenceReader;
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+import java.util.function.LongConsumer;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class DiskUsageStatisticUtil implements Closeable {
+
+ protected static final Logger logger = LoggerFactory.getLogger(DiskUsageStatisticUtil.class);
+ protected Queue<TsFileResource> resourcesWithReadLock;
+ protected final long timePartition;
+ protected final Iterator<TsFileResource> iterator;
+ protected final LongConsumer timeSeriesMetadataIoSizeRecorder;
+ protected final LongConsumer timeSeriesMetadataCountRecorder;
+
+ public DiskUsageStatisticUtil(
+ TsFileManager tsFileManager,
+ long timePartition,
+ Optional<FragmentInstanceContext> fragmentInstanceContext) {
+ this.timePartition = timePartition;
+ this.timeSeriesMetadataIoSizeRecorder =
+ fragmentInstanceContext
+ .<LongConsumer>map(
+ context ->
+ context.getQueryStatistics().getLoadTimeSeriesMetadataActualIOSize()::addAndGet)
+ .orElse(null);
+ this.timeSeriesMetadataCountRecorder =
+ fragmentInstanceContext
+ .<LongConsumer>map(
+ context ->
+ context.getQueryStatistics().getLoadTimeSeriesMetadataFromDiskCount()
+ ::addAndGet)
+ .orElse(null);
+ List<TsFileResource> seqResources = tsFileManager.getTsFileListSnapshot(timePartition, true);
+ List<TsFileResource> unseqResources = tsFileManager.getTsFileListSnapshot(timePartition, false);
+ List<TsFileResource> resources =
+ Stream.concat(seqResources.stream(), unseqResources.stream()).collect(Collectors.toList());
+ acquireReadLocks(resources);
+ iterator = resourcesWithReadLock.iterator();
+ }
+
+ public long getTimePartition() {
+ return timePartition;
+ }
+
+ public boolean hasNextFile() {
+ return iterator.hasNext();
+ }
+
+ protected void acquireReadLocks(List<TsFileResource> resources) {
+ this.resourcesWithReadLock = new LinkedList<>();
+ try {
+ for (TsFileResource resource : resources) {
+ if (!resource.isClosed()) {
+ continue;
+ }
+ resource.readLock();
+ if (resource.isDeleted() || !resource.isClosed()) {
+ resource.readUnlock();
+ continue;
+ }
+ resourcesWithReadLock.add(resource);
+ }
+ } catch (Exception e) {
+ releaseReadLocks();
+ throw e;
+ }
+ }
+
+ protected void releaseReadLocks() {
+ if (resourcesWithReadLock == null) {
+ return;
+ }
+ for (TsFileResource resource : resourcesWithReadLock) {
+ resource.readUnlock();
+ }
+ resourcesWithReadLock = null;
+ }
+
+ public void calculateNextFile() {
+ TsFileResource tsFileResource = iterator.next();
+ if (tsFileResource.isDeleted() || calculateWithoutOpenFile(tsFileResource)) {
+ iterator.remove();
+ tsFileResource.readUnlock();
+ return;
+ }
+ FileReaderManager.getInstance().increaseFileReaderReference(tsFileResource, true);
+ try {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance()
+ .get(
+ tsFileResource.getTsFilePath(),
+ tsFileResource.getTsFileID(),
+ true,
+ timeSeriesMetadataIoSizeRecorder);
+ calculateNextFile(tsFileResource, reader);
+ } catch (Exception e) {
+ logger.error("Failed to scan file {}", tsFileResource.getTsFile().getAbsolutePath(), e);
+ } finally {
+ // this operation including readUnlock
+ FileReaderManager.getInstance().decreaseFileReaderReference(tsFileResource, true);
+ iterator.remove();
+ }
+ }
+
+ protected abstract boolean calculateWithoutOpenFile(TsFileResource tsFileResource);
+
+ protected abstract void calculateNextFile(
+ TsFileResource tsFileResource, TsFileSequenceReader reader)
+ throws IOException, IllegalPathException;
+
+ protected static Offsets calculateStartOffsetOfChunkGroupAndTimeseriesMetadata(
+ TsFileSequenceReader reader,
+ MetadataIndexNode firstMeasurementNodeOfCurrentDevice,
+ Pair<IDeviceID, Boolean> deviceIsAlignedPair,
+ long rootMeasurementNodeStartOffset,
+ LongConsumer timeSeriesMetadataCountRecorder,
+ LongConsumer timeSeriesMetadataIoSizeRecorder)
+ throws IOException {
+ int chunkGroupHeaderSize =
+ new ChunkGroupHeader(deviceIsAlignedPair.getLeft()).getSerializedSize();
+ if (deviceIsAlignedPair.getRight()) {
+ Pair<Long, Long> timeseriesMetadataOffsetPair =
+ getTimeColumnMetadataOffset(
+ reader, firstMeasurementNodeOfCurrentDevice, timeSeriesMetadataIoSizeRecorder);
+ IChunkMetadata firstChunkMetadata =
+ reader
+ .getChunkMetadataListByTimeseriesMetadataOffset(
+ timeseriesMetadataOffsetPair.getLeft(), timeseriesMetadataOffsetPair.getRight())
+ .get(0);
+ if (timeSeriesMetadataCountRecorder != null) {
+ timeSeriesMetadataIoSizeRecorder.accept(
Review Comment:
Variable [timeSeriesMetadataIoSizeRecorder](1) may be null at this access as suggested by [this](2) null guard.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/tsfile/TsFileID.java:
##########
@@ -26,6 +26,8 @@
public class TsFileID {
+ public static final long SHALLOW_SIZE = TsFileID.SHALLOW_SIZE;
Review Comment:
Circular reference detected: `TsFileID.SHALLOW_SIZE` is defined as `TsFileID.SHALLOW_SIZE`. This will result in uninitialized constant value (likely 0) and incorrect memory estimation. The field should reference `RamUsageEstimator.shallowSizeOfInstance(TsFileID.class)` instead.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java:
##########
@@ -2208,6 +2208,16 @@ public synchronized void loadHotModifiedProps(TrimProperties properties)
conf.setMaxRowsInCteBuffer(maxRowsInCteBuffer);
}
+ // max sub-task num for information table scan
+ int maxSubTaskNumForInformationTableScan =
+ Integer.parseInt(
+ properties.getProperty(
+ "max_sub_task_num_for_information_table_scan",
+ Integer.toString(conf.getMaxSubTaskNumForInformationTableScan())));
+ if (maxSubTaskNumForInformationTableScan > 0) {
+ conf.setMaxRowsInCteBuffer(maxSubTaskNumForInformationTableScan);
Review Comment:
Configuration property mismatch: The method sets `maxRowsInCteBuffer` instead of `maxSubTaskNumForInformationTableScan`. This will cause the wrong configuration property to be updated when loading hot-modified properties for `max_sub_task_num_for_information_table_scan`. The correct call should be `conf.setMaxSubTaskNumForInformationTableScan(maxSubTaskNumForInformationTableScan)`.
```suggestion
conf.setMaxSubTaskNumForInformationTableScan(maxSubTaskNumForInformationTableScan);
```
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
Review Comment:
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/DiskUsageStatisticUtil.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.db.queryengine.execution.fragment.FragmentInstanceContext;
+import org.apache.iotdb.db.storageengine.dataregion.read.control.FileReaderManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+
+import org.apache.tsfile.file.IMetadataIndexEntry;
+import org.apache.tsfile.file.header.ChunkGroupHeader;
+import org.apache.tsfile.file.metadata.IChunkMetadata;
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.file.metadata.MetadataIndexNode;
+import org.apache.tsfile.file.metadata.enums.MetadataIndexNodeType;
+import org.apache.tsfile.read.TsFileSequenceReader;
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Queue;
+import java.util.function.LongConsumer;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public abstract class DiskUsageStatisticUtil implements Closeable {
+
+ protected static final Logger logger = LoggerFactory.getLogger(DiskUsageStatisticUtil.class);
+ protected Queue<TsFileResource> resourcesWithReadLock;
+ protected final long timePartition;
+ protected final Iterator<TsFileResource> iterator;
+ protected final LongConsumer timeSeriesMetadataIoSizeRecorder;
+ protected final LongConsumer timeSeriesMetadataCountRecorder;
+
+ public DiskUsageStatisticUtil(
+ TsFileManager tsFileManager,
+ long timePartition,
+ Optional<FragmentInstanceContext> fragmentInstanceContext) {
+ this.timePartition = timePartition;
+ this.timeSeriesMetadataIoSizeRecorder =
+ fragmentInstanceContext
+ .<LongConsumer>map(
+ context ->
+ context.getQueryStatistics().getLoadTimeSeriesMetadataActualIOSize()::addAndGet)
+ .orElse(null);
+ this.timeSeriesMetadataCountRecorder =
+ fragmentInstanceContext
+ .<LongConsumer>map(
+ context ->
+ context.getQueryStatistics().getLoadTimeSeriesMetadataFromDiskCount()
+ ::addAndGet)
+ .orElse(null);
+ List<TsFileResource> seqResources = tsFileManager.getTsFileListSnapshot(timePartition, true);
+ List<TsFileResource> unseqResources = tsFileManager.getTsFileListSnapshot(timePartition, false);
+ List<TsFileResource> resources =
+ Stream.concat(seqResources.stream(), unseqResources.stream()).collect(Collectors.toList());
+ acquireReadLocks(resources);
+ iterator = resourcesWithReadLock.iterator();
+ }
+
+ public long getTimePartition() {
+ return timePartition;
+ }
+
+ public boolean hasNextFile() {
+ return iterator.hasNext();
+ }
+
+ protected void acquireReadLocks(List<TsFileResource> resources) {
+ this.resourcesWithReadLock = new LinkedList<>();
+ try {
+ for (TsFileResource resource : resources) {
+ if (!resource.isClosed()) {
+ continue;
+ }
+ resource.readLock();
+ if (resource.isDeleted() || !resource.isClosed()) {
+ resource.readUnlock();
+ continue;
+ }
+ resourcesWithReadLock.add(resource);
+ }
+ } catch (Exception e) {
+ releaseReadLocks();
+ throw e;
+ }
+ }
+
+ protected void releaseReadLocks() {
+ if (resourcesWithReadLock == null) {
+ return;
+ }
+ for (TsFileResource resource : resourcesWithReadLock) {
+ resource.readUnlock();
+ }
+ resourcesWithReadLock = null;
+ }
+
+ public void calculateNextFile() {
+ TsFileResource tsFileResource = iterator.next();
+ if (tsFileResource.isDeleted() || calculateWithoutOpenFile(tsFileResource)) {
+ iterator.remove();
+ tsFileResource.readUnlock();
+ return;
+ }
+ FileReaderManager.getInstance().increaseFileReaderReference(tsFileResource, true);
+ try {
+ TsFileSequenceReader reader =
+ FileReaderManager.getInstance()
+ .get(
+ tsFileResource.getTsFilePath(),
+ tsFileResource.getTsFileID(),
+ true,
+ timeSeriesMetadataIoSizeRecorder);
+ calculateNextFile(tsFileResource, reader);
+ } catch (Exception e) {
+ logger.error("Failed to scan file {}", tsFileResource.getTsFile().getAbsolutePath(), e);
+ } finally {
+ // this operation including readUnlock
+ FileReaderManager.getInstance().decreaseFileReaderReference(tsFileResource, true);
+ iterator.remove();
+ }
+ }
+
+ protected abstract boolean calculateWithoutOpenFile(TsFileResource tsFileResource);
+
+ protected abstract void calculateNextFile(
+ TsFileResource tsFileResource, TsFileSequenceReader reader)
+ throws IOException, IllegalPathException;
+
+ protected static Offsets calculateStartOffsetOfChunkGroupAndTimeseriesMetadata(
+ TsFileSequenceReader reader,
+ MetadataIndexNode firstMeasurementNodeOfCurrentDevice,
+ Pair<IDeviceID, Boolean> deviceIsAlignedPair,
+ long rootMeasurementNodeStartOffset,
+ LongConsumer timeSeriesMetadataCountRecorder,
+ LongConsumer timeSeriesMetadataIoSizeRecorder)
+ throws IOException {
+ int chunkGroupHeaderSize =
+ new ChunkGroupHeader(deviceIsAlignedPair.getLeft()).getSerializedSize();
+ if (deviceIsAlignedPair.getRight()) {
+ Pair<Long, Long> timeseriesMetadataOffsetPair =
+ getTimeColumnMetadataOffset(
+ reader, firstMeasurementNodeOfCurrentDevice, timeSeriesMetadataIoSizeRecorder);
+ IChunkMetadata firstChunkMetadata =
+ reader
+ .getChunkMetadataListByTimeseriesMetadataOffset(
+ timeseriesMetadataOffsetPair.getLeft(), timeseriesMetadataOffsetPair.getRight())
+ .get(0);
+ if (timeSeriesMetadataCountRecorder != null) {
+ timeSeriesMetadataIoSizeRecorder.accept(
+ timeseriesMetadataOffsetPair.getRight() - timeseriesMetadataOffsetPair.getLeft());
+ timeSeriesMetadataCountRecorder.accept(1);
+ }
+ return new Offsets(
+ firstChunkMetadata.getOffsetOfChunkHeader() - chunkGroupHeaderSize,
+ timeseriesMetadataOffsetPair.getLeft(),
+ rootMeasurementNodeStartOffset);
+ } else {
+ Map<String, Pair<List<IChunkMetadata>, Pair<Long, Long>>> timeseriesMetadataOffsetByDevice =
+ reader.getTimeseriesMetadataOffsetByDevice(
+ firstMeasurementNodeOfCurrentDevice, Collections.emptySet(), true);
+ long minTimeseriesMetadataOffset = 0;
+ long minChunkOffset = Long.MAX_VALUE;
+ for (Map.Entry<String, Pair<List<IChunkMetadata>, Pair<Long, Long>>> entry :
+ timeseriesMetadataOffsetByDevice.entrySet()) {
+ minTimeseriesMetadataOffset =
+ minTimeseriesMetadataOffset == 0
+ ? entry.getValue().getRight().getLeft()
+ : minTimeseriesMetadataOffset;
+ if (timeSeriesMetadataIoSizeRecorder != null) {
+ timeSeriesMetadataIoSizeRecorder.accept(
+ entry.getValue().getRight().getRight() - entry.getValue().getRight().getLeft());
+ timeSeriesMetadataCountRecorder.accept(1);
Review Comment:
Variable [timeSeriesMetadataCountRecorder](1) may be null at this access as suggested by [this](2) null guard.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
Review Comment:
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
Review Comment:
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
Review Comment:
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
Review Comment:
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
Review Comment:
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
Review Comment:
Method testRecoverWriter3 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
Review Comment:
Potential uncaught 'java.lang.NumberFormatException'.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter5() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "2"
+ + TsFileTableDiskUsageCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "2"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
Review Comment:
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/ShowDiskUsageOperator.java:
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.source;
+
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeaderFactory;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.utils.StorageEngineTimePartitionIterator;
+import org.apache.iotdb.db.storageengine.dataregion.utils.TreeDiskUsageStatisticUtil;
+
+import org.apache.tsfile.common.conf.TSFileConfig;
+import org.apache.tsfile.common.conf.TSFileDescriptor;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.read.common.block.TsBlockBuilder;
+import org.apache.tsfile.read.filter.basic.Filter;
+import org.apache.tsfile.read.reader.series.PaginationController;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.util.NoSuchElementException;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+public class ShowDiskUsageOperator implements SourceOperator {
+
+ private static final long INSTANCE_SIZE =
+ RamUsageEstimator.shallowSizeOfInstance(ShowDiskUsageOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final PlanNodeId sourceId;
+ private final PartialPath pathPattern;
+ private final StorageEngineTimePartitionIterator timePartitionIterator;
+ private final PaginationController paginationController;
+ private final TsBlockBuilder tsBlockBuilder =
+ new TsBlockBuilder(DatasetHeaderFactory.getShowDiskUsageHeader().getRespDataTypes());
+ private TreeDiskUsageStatisticUtil statisticUtil;
+ private boolean allConsumed = false;
+
+ public ShowDiskUsageOperator(
+ OperatorContext operatorContext,
+ PlanNodeId sourceId,
+ PartialPath pathPattern,
+ Filter pushDownFilter,
+ PaginationController paginationController) {
+ this.operatorContext = operatorContext;
+ this.sourceId = sourceId;
+ this.pathPattern = pathPattern;
+ this.paginationController = paginationController;
+ this.timePartitionIterator =
+ new StorageEngineTimePartitionIterator(
+ Optional.of(
+ dataRegion -> {
+ String databaseName = dataRegion.getDatabaseName();
+ return !dataRegion.isTableModel()
+ && pathPattern.matchPrefixPath(new PartialPath(databaseName));
+ }),
+ Optional.of(
+ (dataRegion, timePartition) -> {
+ if (pushDownFilter != null) {
+ Object[] row = new Object[4];
+ row[0] = new Binary(dataRegion.getDatabaseName(), TSFileConfig.STRING_CHARSET);
+ row[1] = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ row[2] = Integer.parseInt(dataRegion.getDataRegionIdString());
Review Comment:
Potential uncaught 'java.lang.NumberFormatException'.
```suggestion
try {
row[2] = Integer.parseInt(dataRegion.getDataRegionIdString());
} catch (NumberFormatException e) {
// Skip this data region if its ID is not a valid integer
return false;
}
```
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
Review Comment:
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/object/IObjectTableSizeCacheReader.java:
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.object;
+
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+
+import java.io.IOException;
+
+public interface IObjectTableSizeCacheReader extends AutoCloseable {
+ boolean loadObjectFileTableSize(
+ DataRegionTableSizeQueryContext dataRegionContext, long startTime, long maxRunTime)
+ throws IOException;
+
+ void close();
Review Comment:
This method overrides [AutoCloseable.close](1); it is advisable to add an Override annotation.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
Review Comment:
This method overrides [AbstractCompactionTest.setUp](1); it is advisable to add an Override annotation.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter5() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "2"
+ + TsFileTableDiskUsageCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "2"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
Review Comment:
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/InformationSchemaContentSupplierFactory.java:
##########
@@ -1165,7 +1215,297 @@ public boolean hasNext() {
}
}
- private abstract static class TsBlockSupplier implements Iterator<TsBlock> {
+ private static class TableDiskUsageSupplier implements IInformationSchemaContentSupplier {
+ private final List<TSDataType> dataTypes;
+ private final Map<String, List<TTableInfo>> databaseTableInfoMap;
+ private final Filter pushDownFilter;
+ private final PaginationController paginationController;
+ private final OperatorContext operatorContext;
+
+ private DataRegion currentDataRegion;
+ private boolean currentDatabaseOnlyHasOneTable;
+
+ private TableDiskUsageCacheReader currentDataRegionCacheReader;
+ private DataRegionTableSizeQueryContext currentDataRegionTableSizeQueryContext;
+
+ private final StorageEngineTimePartitionIterator dataRegionIterator;
+
+ private long prepareCacheReaderCostInNS = 0;
+ private long loadObjectFileCostInNS = 0;
+ private long prepareCachedTsFileIDCostInNS = 0;
+ private long checkAllFilesInTsFileManagerCostInNS = 0;
+ private long readTsFileCacheValueFilesCostInNS = 0;
+
+ private TableDiskUsageSupplier(
+ final List<TSDataType> dataTypes,
+ final UserEntity userEntity,
+ final Filter pushDownFilter,
+ final PaginationController paginationController,
+ final OperatorContext operatorContext,
+ final List<Integer> regionsForCurrentSubTask)
+ throws TException, ClientManagerException {
+ this.dataTypes = dataTypes;
+ this.pushDownFilter = pushDownFilter;
+ this.paginationController = paginationController;
+ this.operatorContext = operatorContext;
+ AuthorityChecker.getAccessControl().checkUserGlobalSysPrivilege(userEntity);
+ this.databaseTableInfoMap =
+ operatorContext.getInstanceContext().getDataNodeQueryContext().getDatabaseTableInfoMap();
+ Set<Integer> regions = new HashSet<>(regionsForCurrentSubTask);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.REGIONS_OF_CURRENT_SUB_TASK, regionsForCurrentSubTask.toString());
+ this.dataRegionIterator =
+ new StorageEngineTimePartitionIterator(
+ Optional.of(
+ dataRegion -> {
+ List<TTableInfo> tTableInfos =
+ databaseTableInfoMap.get(dataRegion.getDatabaseName());
+ if (tTableInfos == null || tTableInfos.isEmpty()) {
+ return false;
+ }
+ if (!regions.contains(dataRegion.getDataRegionId().getId())) {
+ return false;
+ }
+ currentDataRegionTableSizeQueryContext =
+ new DataRegionTableSizeQueryContext(
+ false, operatorContext.getInstanceContext());
+ return true;
+ }),
+ Optional.empty());
+ }
+
+ @Override
+ public boolean hasNext() {
+ boolean result = hasNextInternal();
+ if (!result) {
+ updateSpecifiedInfo();
+ }
+ return result;
+ }
+
+ private boolean hasNextInternal() {
+ if (currentDataRegionCacheReader != null) {
+ return true;
+ }
+ if (!paginationController.hasCurLimit()) {
+ return false;
+ }
+ try {
+ while (dataRegionIterator.nextDataRegion()) {
+ currentDataRegion = dataRegionIterator.currentDataRegion();
+ for (Long timePartition : currentDataRegion.getTsFileManager().getTimePartitions()) {
+ Map<String, Long> tablesToScan = getTablesToScan(currentDataRegion, timePartition);
+ if (!tablesToScan.isEmpty()) {
+ currentDataRegionTableSizeQueryContext.addTimePartition(
+ timePartition, new TimePartitionTableSizeQueryContext(tablesToScan));
+ }
+ }
+ if (currentDataRegionTableSizeQueryContext.isEmpty()) {
+ continue;
+ }
+ currentDataRegionCacheReader =
+ new TableDiskUsageCacheReader(
+ currentDataRegion,
+ currentDataRegionTableSizeQueryContext,
+ currentDatabaseOnlyHasOneTable);
+ return true;
+ }
+ } catch (Exception e) {
+ closeDataRegionReader();
+ throw new IoTDBRuntimeException(
+ e.getMessage(), e, TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode());
+ }
+ return false;
+ }
+
+ private void updateSpecifiedInfo() {
+ if (operatorContext
+ .getSpecifiedInfo()
+ .containsKey(PlanGraphPrinter.PREPARE_CACHE_READER_COST)) {
+ return;
+ }
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.PREPARE_CACHE_READER_COST,
+ TimeUnit.NANOSECONDS.toMillis(prepareCacheReaderCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.LOAD_OBJECT_FILE_COST,
+ TimeUnit.NANOSECONDS.toMillis(loadObjectFileCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.PREPARE_CACHED_TSFILE_ID_COST,
+ TimeUnit.NANOSECONDS.toMillis(prepareCachedTsFileIDCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.CHECK_ALL_FILES_IN_TSFILE_MANAGER_COST,
+ TimeUnit.NANOSECONDS.toMillis(checkAllFilesInTsFileManagerCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.READ_TSFILE_CACHE_VALUE_FILES_COST,
+ TimeUnit.NANOSECONDS.toMillis(readTsFileCacheValueFilesCostInNS)
+ + IoTDBConstant.SPACE
+ + RpcUtils.MILLISECOND);
+ }
+
+ private Map<String, Long> getTablesToScan(DataRegion dataRegion, long timePartition) {
+ String databaseName = dataRegion.getDatabaseName();
+ List<TTableInfo> tTableInfos = databaseTableInfoMap.get(databaseName);
+ if (tTableInfos == null || tTableInfos.isEmpty()) {
+ return Collections.emptyMap();
+ }
+
+ Map<String, Long> tablesToScan = new TreeMap<>();
+ int totalValidTableCount = 0;
+ for (TTableInfo tTableInfo : tTableInfos) {
+ if (tTableInfo.getType() != TableType.BASE_TABLE.ordinal()) {
+ continue;
+ }
+ totalValidTableCount++;
+ if (pushDownFilter != null) {
+ Object[] row = new Object[5];
+ row[0] = new Binary(dataRegion.getDatabaseName(), TSFileConfig.STRING_CHARSET);
+ row[1] = new Binary(tTableInfo.getTableName(), TSFileConfig.STRING_CHARSET);
+ row[2] = IoTDBDescriptor.getInstance().getConfig().getDataNodeId();
+ row[3] = dataRegion.getDataRegionId().getId();
+ row[4] = timePartition;
+ if (!pushDownFilter.satisfyRow(0, row)) {
+ continue;
+ }
+ }
+ if (!paginationController.hasCurLimit()) {
+ break;
+ }
+ if (paginationController.hasCurOffset()) {
+ paginationController.consumeOffset();
+ continue;
+ }
+ paginationController.consumeLimit();
+ tablesToScan.put(tTableInfo.getTableName(), 0L);
+ }
+ currentDatabaseOnlyHasOneTable = totalValidTableCount == 1;
+ return tablesToScan;
+ }
+
+ @Override
+ public TsBlock next() {
+ if (!hasNext()) {
+ throw new NoSuchElementException();
+ }
+
+ long maxRuntime = OperatorContext.getMaxRunTime().roundTo(TimeUnit.NANOSECONDS);
+ long start = System.nanoTime();
+ long prevStageEndTime = start;
+
+ try {
+ try {
+ if (!currentDataRegionCacheReader.prepareCacheReader(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ prepareCacheReaderCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.loadObjectFileTableSizeCache(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ loadObjectFileCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.prepareCachedTsFileIDKeys(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ prepareCachedTsFileIDCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.checkAllFilesInTsFileManager(start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ long now = System.nanoTime();
+ checkAllFilesInTsFileManagerCostInNS += now - prevStageEndTime;
+ prevStageEndTime = now;
+ }
+
+ try {
+ if (!currentDataRegionCacheReader.readCacheValueFilesAndUpdateResultMap(
+ start, maxRuntime)) {
+ return null;
+ }
+ } finally {
+ readTsFileCacheValueFilesCostInNS += System.nanoTime() - prevStageEndTime;
+ }
+
+ return buildTsBlock();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ return null;
+ } catch (Exception e) {
+ throw new IoTDBRuntimeException(
+ e.getMessage(), e, TSStatusCode.INTERNAL_SERVER_ERROR.getStatusCode());
+ }
+ }
+
+ private TsBlock buildTsBlock() {
+ TsBlockBuilder builder = new TsBlockBuilder(dataTypes);
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ currentDataRegionTableSizeQueryContext
+ .getTimePartitionTableSizeQueryContextMap()
+ .entrySet()) {
+ long timePartition = entry.getKey();
+ for (Map.Entry<String, Long> tableSizeEntry :
+ entry.getValue().getTableSizeResultMap().entrySet()) {
+ String tableName = tableSizeEntry.getKey();
+ long size = tableSizeEntry.getValue();
+ builder.getTimeColumnBuilder().writeLong(0);
+ ColumnBuilder[] columns = builder.getValueColumnBuilders();
+
+ columns[0].writeBinary(
+ new Binary(currentDataRegion.getDatabaseName(), TSFileConfig.STRING_CHARSET));
+ columns[1].writeBinary(new Binary(tableName, TSFileConfig.STRING_CHARSET));
+ columns[2].writeInt(IoTDBDescriptor.getInstance().getConfig().getDataNodeId());
+ columns[3].writeInt(Integer.parseInt(currentDataRegion.getDataRegionIdString()));
Review Comment:
Potential uncaught 'java.lang.NumberFormatException'.
```suggestion
int dataRegionId;
try {
dataRegionId = Integer.parseInt(currentDataRegion.getDataRegionIdString());
} catch (NumberFormatException e) {
throw new IoTDBRuntimeException(
"Invalid data region id string: "
+ currentDataRegion.getDataRegionIdString()
+ " for database "
+ currentDataRegion.getDatabaseName(),
e);
}
columns[3].writeInt(dataRegionId);
```
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TableDiskUsageTest.java:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.tablemodel.CompactionTableModelTestFileWriter;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TableDiskUsageCache;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TableDiskUsageCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.read.common.TimeRange;
+import org.apache.tsfile.utils.Pair;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class TableDiskUsageTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
Review Comment:
This method overrides [AbstractCompactionTest.setUp](1); it is advisable to add an Override annotation.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter5() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "2"
+ + TsFileTableDiskUsageCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "2"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ tempKeyFile2.createNewFile();
Review Comment:
Method testRecoverWriter5 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/table/InformationSchema.java:
##########
@@ -21,17 +21,25 @@
import org.apache.iotdb.commons.schema.column.ColumnHeaderConstant;
import org.apache.iotdb.commons.schema.table.column.AttributeColumnSchema;
+import org.apache.iotdb.commons.schema.table.column.FieldColumnSchema;
import org.apache.iotdb.commons.schema.table.column.TagColumnSchema;
+import com.google.common.collect.ImmutableSet;
import org.apache.tsfile.enums.TSDataType;
+import java.util.Collections;
import java.util.HashMap;
+import java.util.HashSet;
import java.util.Locale;
import java.util.Map;
+import java.util.Set;
public class InformationSchema {
public static final String INFORMATION_DATABASE = "information_schema";
private static final Map<String, TsTable> schemaTables = new HashMap<>();
+ private static final Map<String, Set<String>> columnsThatSupportPushDownPredicate =
+ new HashMap<>();
+ private static final Set<String> tablesThatSupportPushDownLimitOffset = new HashSet<>();
Review Comment:
The contents of this container are never accessed.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/tsfile/TsFileTableSizeIndexFileWriter.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile;
+
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.channels.FileChannel;
+import java.nio.file.StandardOpenOption;
+import java.util.Map;
+
+import static org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET;
+import static org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_REDIRECT;
+
+public class TsFileTableSizeIndexFileWriter {
+ private final int regionId;
+ private final File currentKeyIndexFile;
+ private final File currentValueIndexFile;
+ private FileOutputStream keyFileOutputStream;
+ private FileOutputStream valueFileOutputStream;
+ private BufferedOutputStream keyBufferedOutputStream;
+ private BufferedOutputStream valueBufferedOutputStream;
+ private long keyFileSize;
+ private long valueFileSize;
+
+ public TsFileTableSizeIndexFileWriter(
+ int regionId, File currentKeyIndexFile, File currentValueIndexFile) throws IOException {
+ this(regionId, currentKeyIndexFile, currentValueIndexFile, true);
+ }
+
+ public TsFileTableSizeIndexFileWriter(
+ int regionId, File currentKeyIndexFile, File currentValueIndexFile, boolean recover)
+ throws IOException {
+ this.regionId = regionId;
+ this.currentKeyIndexFile = currentKeyIndexFile;
+ this.currentValueIndexFile = currentValueIndexFile;
+ currentKeyIndexFile.createNewFile();
+ currentValueIndexFile.createNewFile();
Review Comment:
Method TsFileTableSizeIndexFileWriter ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/utils/tableDiskUsageCache/tsfile/TsFileTableSizeIndexFileWriter.java:
##########
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile;
+
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.nio.channels.FileChannel;
+import java.nio.file.StandardOpenOption;
+import java.util.Map;
+
+import static org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET;
+import static org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_REDIRECT;
+
+public class TsFileTableSizeIndexFileWriter {
+ private final int regionId;
+ private final File currentKeyIndexFile;
+ private final File currentValueIndexFile;
+ private FileOutputStream keyFileOutputStream;
+ private FileOutputStream valueFileOutputStream;
+ private BufferedOutputStream keyBufferedOutputStream;
+ private BufferedOutputStream valueBufferedOutputStream;
+ private long keyFileSize;
+ private long valueFileSize;
+
+ public TsFileTableSizeIndexFileWriter(
+ int regionId, File currentKeyIndexFile, File currentValueIndexFile) throws IOException {
+ this(regionId, currentKeyIndexFile, currentValueIndexFile, true);
+ }
+
+ public TsFileTableSizeIndexFileWriter(
+ int regionId, File currentKeyIndexFile, File currentValueIndexFile, boolean recover)
+ throws IOException {
+ this.regionId = regionId;
+ this.currentKeyIndexFile = currentKeyIndexFile;
+ this.currentValueIndexFile = currentValueIndexFile;
+ currentKeyIndexFile.createNewFile();
Review Comment:
Method TsFileTableSizeIndexFileWriter ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
Review Comment:
Method testRecoverWriter2 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TableDiskUsageTest.java:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.tablemodel.CompactionTableModelTestFileWriter;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TableDiskUsageCache;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TableDiskUsageCacheReader;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.read.common.TimeRange;
+import org.apache.tsfile.utils.Pair;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class TableDiskUsageTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ TableDiskUsageCache.getInstance().ensureRunning();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionId()).thenReturn(new DataRegionId(0));
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ TableDiskUsageCache.getInstance().registerRegion(mockDataRegion);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
Review Comment:
This method overrides [AbstractCompactionTest.tearDown](1); it is advisable to add an Override annotation.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##########
@@ -1994,8 +2005,9 @@ public void deleteFolder(String systemDir) {
"{} will close all files for deleting data folder {}",
databaseName + "-" + dataRegionIdString,
systemDir);
- FileTimeIndexCacheRecorder.getInstance()
- .removeFileTimeIndexCache(Integer.parseInt(dataRegionIdString));
+ int regionId = Integer.parseInt(dataRegionIdString);
Review Comment:
Potential uncaught 'java.lang.NumberFormatException'.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
Review Comment:
This method overrides [AbstractCompactionTest.tearDown](1); it is advisable to add an Override annotation.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
Review Comment:
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
Review Comment:
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/utils/TsFileTableSizeCacheWriterTest.java:
##########
@@ -0,0 +1,466 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.utils;
+
+import org.apache.iotdb.commons.consensus.DataRegionId;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.storageengine.StorageEngine;
+import org.apache.iotdb.db.storageengine.dataregion.DataRegion;
+import org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileID;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.AbstractTableSizeCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.DataRegionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.TimePartitionTableSizeQueryContext;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableDiskUsageCacheWriter;
+import org.apache.iotdb.db.storageengine.dataregion.utils.tableDiskUsageCache.tsfile.TsFileTableSizeCacheReader;
+
+import org.apache.tsfile.exception.write.WriteProcessException;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.ReadWriteForEncodingUtils;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Files;
+import java.nio.file.StandardOpenOption;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+public class TsFileTableSizeCacheWriterTest extends AbstractCompactionTest {
+
+ private DataRegion mockDataRegion;
+ private TsFileManager mockTsFileManager;
+
+ @Before
+ public void setUp()
+ throws IOException, WriteProcessException, MetadataException, InterruptedException {
+ super.setUp();
+ mockDataRegion = Mockito.mock(DataRegion.class);
+ Mockito.when(mockDataRegion.getDatabaseName()).thenReturn("test");
+ Mockito.when(mockDataRegion.getDataRegionIdString()).thenReturn("0");
+ StorageEngine.getInstance().setDataRegion(new DataRegionId(0), mockDataRegion);
+ mockTsFileManager = new TsFileManager("test", "0", "");
+ Mockito.when(mockDataRegion.getTsFileManager()).thenReturn(mockTsFileManager);
+ }
+
+ @After
+ public void tearDown() throws IOException, StorageEngineException {
+ super.tearDown();
+ StorageEngine.getInstance().deleteDataRegion(new DataRegionId(0));
+ }
+
+ @Test
+ public void testCompactEmptyTargetFile() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 20L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", writer.getKeyFile().getName());
+ Assert.assertEquals("TableSizeValueFile_1", writer.getValueFile().getName());
+ Assert.assertEquals(0, writer.keyFileLength());
+ Assert.assertEquals(0, writer.valueFileLength());
+ writer.close();
+ }
+
+ @Test
+ public void testCompactTargetFile1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource3, false);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource2.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testCompactTargetFile2() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ File oldKeyFile = writer.getKeyFile();
+ File oldValueFile = writer.getValueFile();
+ Assert.assertEquals("TableSizeKeyFile_0", oldKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_0", oldValueFile.getName());
+
+ TsFileResource resource1 = createEmptyFileAndResourceWithName("1-1-0-0.tsfile", 1, true);
+ TsFileResource resource2 = createEmptyFileAndResourceWithName("2-2-0-0.tsfile", 1, true);
+ TsFileResource resource3 = createEmptyFileAndResourceWithName("3-3-0-0.tsfile", 1, false);
+ mockTsFileManager.add(resource1, true);
+ mockTsFileManager.add(resource2, true);
+
+ writer.write(resource1.getTsFileID(), Collections.singletonMap("table1", 10L));
+ writer.write(resource3.getTsFileID(), Collections.singletonMap("table2", 200L));
+ writer.write(resource3.getTsFileID(), resource2.getTsFileID());
+
+ writer.compact();
+
+ File targetKeyFile = writer.getKeyFile();
+ File targetValueFile = writer.getValueFile();
+ Assert.assertFalse(oldKeyFile.exists());
+ Assert.assertFalse(oldValueFile.exists());
+ Assert.assertEquals("TableSizeKeyFile_1", targetKeyFile.getName());
+ Assert.assertEquals("TableSizeValueFile_1", targetValueFile.getName());
+ writer.close();
+
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(
+ targetKeyFile.length(), targetKeyFile, targetValueFile.length(), targetValueFile, 1);
+ reader.openKeyFile();
+ int count = 0;
+ while (reader.hasNextEntryInKeyFile()) {
+ TsFileTableSizeCacheReader.KeyFileEntry keyFileEntry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotEquals(3, keyFileEntry.tsFileID.fileVersion);
+ count++;
+ }
+ reader.closeCurrentFile();
+ Assert.assertEquals(2, count);
+ }
+
+ @Test
+ public void testReadPerformance() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ Assert.assertNotEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertNotEquals(writer.getValueFile().length(), writer.valueFileLength());
+ writer.close();
+ Assert.assertEquals(writer.getKeyFile().length(), writer.keyFileLength());
+ Assert.assertEquals(writer.getValueFile().length(), writer.valueFileLength());
+ File keyFile = writer.getKeyFile();
+ File valueFile = writer.getValueFile();
+ DataRegionTableSizeQueryContext context = new DataRegionTableSizeQueryContext(true);
+ TsFileTableSizeCacheReader reader =
+ new TsFileTableSizeCacheReader(keyFile.length(), keyFile, valueFile.length(), valueFile, 0);
+ List<Pair<TsFileID, Long>> offsets = new ArrayList<>();
+ long start = System.currentTimeMillis();
+ reader.openKeyFile();
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10000; j++) {
+ Assert.assertTrue(reader.hasNextEntryInKeyFile());
+ TsFileTableSizeCacheReader.KeyFileEntry entry = reader.readOneEntryFromKeyFile();
+ Assert.assertNotNull(entry);
+ Assert.assertNull(entry.originTsFileID);
+ Assert.assertEquals(new TsFileID(0, i, j, j, 0L), entry.tsFileID);
+ Assert.assertTrue(entry.offset >= 0);
+ offsets.add(new Pair<>(entry.tsFileID, entry.offset));
+ }
+ }
+ Assert.assertFalse(reader.hasNextEntryInKeyFile());
+ reader.closeCurrentFile();
+ reader.openValueFile();
+ reader.readFromValueFile(offsets.iterator(), context, System.nanoTime(), Long.MAX_VALUE);
+ reader.closeCurrentFile();
+ System.out.println("cost: " + (System.currentTimeMillis() - start) + "ms");
+ System.out.println("keyFileLength: " + keyFile.length());
+ System.out.println("valueFileLength: " + valueFile.length());
+
+ Assert.assertEquals(10, context.getTimePartitionTableSizeQueryContextMap().size());
+ for (Map.Entry<Long, TimePartitionTableSizeQueryContext> entry :
+ context.getTimePartitionTableSizeQueryContextMap().entrySet()) {
+ Map<String, Long> tableSizeResultMap = entry.getValue().getTableSizeResultMap();
+ for (Map.Entry<String, Long> tableSizeEntry : tableSizeResultMap.entrySet()) {
+ int i = Integer.parseInt(tableSizeEntry.getKey().substring(5));
+ Assert.assertEquals(Long.valueOf(i * 10000L), tableSizeEntry.getValue());
+ }
+ }
+ }
+
+ @Test
+ public void testRecoverWriter1() throws IOException {
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ for (int i = 1; i <= 10; i++) {
+ for (int j = 1; j <= 10; j++) {
+ TsFileResource resource =
+ createEmptyFileAndResourceWithName(j + "-" + j + "-0-0.tsfile", i, true);
+ writer.write(resource.getTsFileID(), generateTableSizeMap(i));
+ }
+ }
+ writer.close();
+ File keyFile = writer.getKeyFile();
+ long keyFileValidLength = keyFile.length();
+ File valueFile = writer.getValueFile();
+ long valueFileValidLength = valueFile.length();
+ Files.write(keyFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Files.write(valueFile.toPath(), new byte[] {1, 2, 3, 4}, StandardOpenOption.APPEND);
+ Assert.assertEquals(keyFileValidLength + 4, keyFile.length());
+ Assert.assertEquals(valueFileValidLength + 4, valueFile.length());
+
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ Files.write(keyFile.toPath(), new byte[] {0, 0, 0, 0}, StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ ByteBuffer buffer =
+ ByteBuffer.allocate(TsFileTableDiskUsageCacheWriter.KEY_FILE_OFFSET_RECORD_LENGTH);
+ TsFileID tsFileID = new TsFileID(0, 1, 2, 3, 4);
+ ReadWriteIOUtils.write(TsFileTableDiskUsageCacheWriter.KEY_FILE_RECORD_TYPE_OFFSET, buffer);
+ ReadWriteIOUtils.write(tsFileID.timePartitionId, buffer);
+ ReadWriteIOUtils.write(tsFileID.timestamp, buffer);
+ ReadWriteIOUtils.write(tsFileID.fileVersion, buffer);
+ ReadWriteIOUtils.write(tsFileID.compactionVersion, buffer);
+ ReadWriteIOUtils.write(valueFileValidLength, buffer);
+ Files.write(
+ keyFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+
+ buffer.clear();
+ ReadWriteForEncodingUtils.writeVarInt(1, buffer);
+ ReadWriteIOUtils.writeVar("table1", buffer);
+ ReadWriteIOUtils.write(10L, buffer);
+ Files.write(
+ valueFile.toPath(),
+ Arrays.copyOf(buffer.array(), buffer.position()),
+ StandardOpenOption.APPEND);
+ writer = new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertEquals(keyFileValidLength, keyFile.length());
+ Assert.assertEquals(valueFileValidLength, valueFile.length());
+ }
+
+ @Test
+ public void testRecoverWriter2() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File tempKeyFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ tempKeyFile2.createNewFile();
+ tempValueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertTrue(keyFile1.exists());
+ Assert.assertTrue(valueFile1.exists());
+ Assert.assertFalse(tempKeyFile2.exists());
+ Assert.assertFalse(tempValueFile2.exists());
+ Assert.assertEquals(keyFile1, writer.getKeyFile());
+ Assert.assertEquals(valueFile1, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter3() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ valueFile2.createNewFile();
+ TsFileTableDiskUsageCacheWriter writer =
+ new TsFileTableDiskUsageCacheWriter(mockDataRegion.getDatabaseName(), 0);
+ writer.close();
+ Assert.assertFalse(keyFile1.exists());
+ Assert.assertFalse(valueFile1.exists());
+ Assert.assertTrue(keyFile2.exists());
+ Assert.assertTrue(valueFile2.exists());
+ Assert.assertEquals(keyFile2, writer.getKeyFile());
+ Assert.assertEquals(valueFile2, writer.getValueFile());
+ }
+
+ @Test
+ public void testRecoverWriter4() throws IOException {
+ File dir =
+ StorageEngine.getDataRegionSystemDir(
+ mockDataRegion.getDatabaseName(), mockDataRegion.getDataRegionIdString());
+ dir.mkdirs();
+ File keyFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "0");
+ File valueFile1 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "0");
+ File keyFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_KEY_FILENAME_PREFIX + "1");
+ File tempValueFile2 =
+ new File(
+ dir,
+ TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX
+ + "1"
+ + AbstractTableSizeCacheWriter.TEMP_CACHE_FILE_SUBFIX);
+ File valueFile2 =
+ new File(dir, TsFileTableDiskUsageCacheWriter.TSFILE_CACHE_VALUE_FILENAME_PREFIX + "1");
+
+ keyFile1.createNewFile();
+ valueFile1.createNewFile();
+ keyFile2.createNewFile();
+ tempValueFile2.createNewFile();
Review Comment:
Method testRecoverWriter4 ignores exceptional return value of File.createNewFile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3852747440
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3857826117
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-4072280138
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3858564456
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3858268214
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-3852927398
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#issuecomment-4053693453
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) **Quality Gate failed**
Failed conditions
 [C Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17169)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement table & tree disk usage statistics [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17169:
URL: https://github.com/apache/iotdb/pull/17169#discussion_r2938526486
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/source/relational/InformationSchemaContentSupplierFactory.java:
##########
@@ -1165,7 +1215,297 @@ public boolean hasNext() {
}
}
- private abstract static class TsBlockSupplier implements Iterator<TsBlock> {
+ private static class TableDiskUsageSupplier implements IInformationSchemaContentSupplier {
+ private final List<TSDataType> dataTypes;
+ private final Map<String, List<TTableInfo>> databaseTableInfoMap;
+ private final Filter pushDownFilter;
+ private final PaginationController paginationController;
+ private final OperatorContext operatorContext;
+
+ private DataRegion currentDataRegion;
+ private boolean currentDatabaseOnlyHasOneTable;
+
+ private TableDiskUsageCacheReader currentDataRegionCacheReader;
+ private DataRegionTableSizeQueryContext currentDataRegionTableSizeQueryContext;
+
+ private final StorageEngineTimePartitionIterator dataRegionIterator;
+
+ private long prepareCacheReaderCostInNS = 0;
+ private long loadObjectFileCostInNS = 0;
+ private long prepareCachedTsFileIDCostInNS = 0;
+ private long checkAllFilesInTsFileManagerCostInNS = 0;
+ private long readTsFileCacheValueFilesCostInNS = 0;
+
+ private TableDiskUsageSupplier(
+ final List<TSDataType> dataTypes,
+ final UserEntity userEntity,
+ final Filter pushDownFilter,
+ final PaginationController paginationController,
+ final OperatorContext operatorContext,
+ final List<Integer> regionsForCurrentSubTask)
+ throws TException, ClientManagerException {
+ this.dataTypes = dataTypes;
+ this.pushDownFilter = pushDownFilter;
+ this.paginationController = paginationController;
+ this.operatorContext = operatorContext;
+ AuthorityChecker.getAccessControl().checkUserGlobalSysPrivilege(userEntity);
+ this.databaseTableInfoMap =
+ operatorContext.getInstanceContext().getDataNodeQueryContext().getDatabaseTableInfoMap();
+ Set<Integer> regions = new HashSet<>(regionsForCurrentSubTask);
+ operatorContext.recordSpecifiedInfo(
+ PlanGraphPrinter.REGIONS_OF_CURRENT_SUB_TASK, regionsForCurrentSubTask.toString());
+ this.dataRegionIterator =
+ new StorageEngineTimePartitionIterator(
+ Optional.of(
+ dataRegion -> {
+ List<TTableInfo> tTableInfos =
+ databaseTableInfoMap.get(dataRegion.getDatabaseName());
+ if (tTableInfos == null || tTableInfos.isEmpty()) {
+ return false;
+ }
+ if (!regions.contains(dataRegion.getDataRegionId())) {
+ return false;
+ }
+ currentDataRegionTableSizeQueryContext =
+ new DataRegionTableSizeQueryContext(
+ false, operatorContext.getInstanceContext());
+ return true;
+ }),
+ Optional.empty());
Review Comment:
add some comments about this weird codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org