You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2026/03/27 06:32:13 UTC
[PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
shuwenwei opened a new pull request, #17372:
URL: https://github.com/apache/iotdb/pull/17372
## Description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4153454964
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate failed**
Failed conditions
 [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4141566715
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [22 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4140727934
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [26 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.2% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4151778103
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [24 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.4% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "shuwenwei (via GitHub)" <gi...@apache.org>.
shuwenwei commented on code in PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#discussion_r3008456902
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/CopyToOptions.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.db.exception.sql.SemanticException;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.CopyToTsFileOptions;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlan;
+
+import org.apache.tsfile.utils.Accountable;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Interface for COPY TO command options.
+ *
+ * <p>This interface defines the configuration for COPY TO operations, including target format,
+ * target table name, column mappings, and memory management. Implementations provide
+ * format-specific validation and inference logic.
+ */
+public interface CopyToOptions extends Accountable {
+
+ /**
+ * Infers and validates options based on query analysis and relation plan.
+ *
+ * <p>This method is called during analysis phase to fill in default values and validate
+ * user-specified options against the query structure. For example, it can infer the target time
+ * column from the query's time column if not explicitly specified.
+ *
+ * @param analysis the query analysis containing metadata about the query
+ * @param queryRelationPlan the logical relation plan of the inner query
+ * @param columnHeaders the column headers from the query result
+ */
+ void infer(Analysis analysis, RelationPlan queryRelationPlan, List<ColumnHeader> columnHeaders);
+
+ /**
+ * Validates the options against the actual column schema.
+ *
+ * <p>This method is called after planning to ensure the specified options (e.g., target columns,
+ * tags) are valid for the given output schema.
+ *
+ * @param columnHeaders the column headers of the query result
+ * @throws SemanticException if validation fails
+ */
+ void check(List<ColumnHeader> columnHeaders);
+
+ /**
+ * Returns the response column headers for the result set.
+ *
+ * <p>These headers describe the columns that will be returned to the client after the COPY TO
+ * operation completes (e.g., file path, row count).
+ *
+ * @return list of column headers for the response
+ */
+ List<ColumnHeader> getRespColumnHeaders();
+
+ /**
+ * Returns the target output format.
+ *
+ * @return the format enum value
+ */
+ Format getFormat();
+
+ /**
+ * Estimates the maximum memory usage in bytes required for writing.
+ *
+ * <p>This is used for memory management and determining whether to flush data to disk.
+ *
+ * @return estimated maximum memory usage in bytes
+ */
+ long estimatedMaxRamBytesInWrite();
+
+ /** Supported output formats for COPY TO command. */
+ enum Format {
+ /** TsFile format output. */
+ TSFILE,
+ }
+
+ class Builder {
+ private CopyToOptions.Format format = CopyToOptions.Format.TSFILE;
+ private String targetTableName = null;
+ private String targetTimeColumn = null;
+ private Set<String> targetTagColumns = null;
+ private long memoryThreshold = 0;
+
+ public Builder withFormat(CopyToOptions.Format format) {
+ this.format = format;
+ return this;
+ }
+
+ public Builder withTargetTableName(String targetTableName) {
+ this.targetTableName = targetTableName;
+ return this;
+ }
+
+ public Builder withTargetTimeColumn(String targetTimeColumn) {
+ this.targetTimeColumn = targetTimeColumn;
+ return this;
+ }
+
+ public Builder withTargetTagColumns(Set<String> targetTagColumns) {
+ this.targetTagColumns = targetTagColumns;
+ return this;
+ }
+
+ public Builder withMemoryThreshold(long memoryThreshold) {
+ if (memoryThreshold <= 0) {
+ throw new SemanticException("The memory threshold must be greater than 0.");
+ }
+ this.memoryThreshold = memoryThreshold;
+ return this;
+ }
+
+ public CopyToOptions build() {
+ switch (format) {
+ case TSFILE:
+ default:
+ return new CopyToTsFileOptions(
+ targetTableName,
+ targetTimeColumn,
+ targetTagColumns,
+ memoryThreshold != 0 ? memoryThreshold : 32 * 1024 * 1024);
Review Comment:
This is not supported in the current API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4160639496
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate failed**
Failed conditions
 [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4153392369
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate failed**
Failed conditions
 [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#discussion_r3007313267
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/parser/AstBuilder.java:
##########
@@ -2155,6 +2161,62 @@ public Node visitRevokeStatement(RelationalSqlParser.RevokeStatementContext ctx)
}
}
+ @Override
+ public Node visitCopyToStatement(RelationalSqlParser.CopyToStatementContext ctx) {
+ RelationalSqlParser.CopyToStatementOptionsContext optionsContext = ctx.copyToStatementOptions();
+ String targetFileName = parseStringLiteral(ctx.fileName.getText());
+ CopyToOptions.Builder copyToOptionsBuilder = new CopyToOptions.Builder();
+ if (optionsContext != null) {
+ for (RelationalSqlParser.CopyToStatementOptionContext context :
+ optionsContext.copyToStatementOption()) {
+ addCopyToOption(copyToOptionsBuilder, context);
+ }
+ }
+ Statement queryNode = null;
+ if (ctx.tableName != null) {
+ QualifiedName qualifiedName = getQualifiedName(ctx.tableName);
+ if (ctx.tableColumns != null) {
+ List<RelationalSqlParser.IdentifierContext> identifierList =
+ ctx.identifierList().identifier();
+ SelectItem[] selectItems = new SelectItem[identifierList.size()];
+ for (int i = 0; i < identifierList.size(); i++) {
+ Identifier identifier = (Identifier) visit(identifierList.get(i));
+ selectItems[i] = new SingleColumn(identifier, identifier);
+ }
+ queryNode = QueryUtil.simpleQuery(selectList(selectItems), table(qualifiedName));
+ } else {
+ queryNode = QueryUtil.simpleQuery(selectList(new AllColumns()), table(qualifiedName));
+ }
+ } else {
+ queryNode = (Statement) visit(ctx.query());
+ }
+ return new CopyTo(queryNode, targetFileName, copyToOptionsBuilder.build());
+ }
+
+ private void addCopyToOption(
+ CopyToOptions.Builder builder, RelationalSqlParser.CopyToStatementOptionContext context) {
+ if (context.FORMAT() != null) {
+ Identifier formatIdentifier = (Identifier) visit(context.identifier());
+ builder.withFormat(CopyToOptions.Format.valueOf(formatIdentifier.getValue().toUpperCase()));
Review Comment:
FORMAT parsing uses CopyToOptions.Format.valueOf(...). If the user provides an unsupported value, this will throw IllegalArgumentException and likely surface as an internal error. Please catch this and rethrow a SemanticException with a clear message listing supported formats.
```suggestion
String formatText = formatIdentifier.getValue().toUpperCase();
try {
builder.withFormat(CopyToOptions.Format.valueOf(formatText));
} catch (IllegalArgumentException e) {
String supportedFormats =
Arrays.stream(CopyToOptions.Format.values())
.map(Enum::name)
.collect(Collectors.joining(", "));
throw new SemanticException(
String.format(
"Unsupported COPY TO format '%s'. Supported formats: %s",
formatIdentifier.getValue(), supportedFormats),
e);
}
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/optimizations/UnaliasSymbolReferences.java:
##########
@@ -388,6 +389,24 @@ public PlanAndMappings visitExplainAnalyze(ExplainAnalyzeNode node, UnaliasConte
mapping);
}
+ @Override
+ public PlanAndMappings visitCopyTo(CopyToNode node, UnaliasContext context) {
+ PlanAndMappings rewrittenSource = node.getChild().accept(this, context);
+ Map<Symbol, Symbol> mapping = new HashMap<>(rewrittenSource.getMappings());
+ SymbolMapper mapper = symbolMapper(mapping);
+ List<Symbol> newChildPermittedOutputs = mapper.map(node.getChildPermittedOutputs());
+ return new PlanAndMappings(
+ new CopyToNode(
+ node.getPlanNodeId(),
+ rewrittenSource.getRoot(),
+ node.getTargetFilePath(),
+ node.getCopyToOptions(),
+ newChildPermittedOutputs,
+ node.getInnerQueryDatasetHeader(),
+ node.getInnerQueryOutputSymbols()),
+ mapping);
Review Comment:
In UnaliasSymbolReferences, CopyToNode rewriting maps childPermittedOutputs but leaves innerQueryOutputSymbols unchanged. If the child plan is rewritten (symbol remapped), these stale symbols can no longer be found in the child's output symbol list, leading to null lookups/NPE when building the column index mapping in TableOperatorGenerator. Map innerQueryOutputSymbols with the same SymbolMapper before constructing the new CopyToNode.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/copyto/IoTDBCopyToTsFileIT.java:
##########
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.relational.it.query.recent.copyto;
+
+import org.apache.iotdb.isession.ITableSession;
+import org.apache.iotdb.isession.SessionDataSet;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.TableClusterIT;
+import org.apache.iotdb.itbase.category.TableLocalStandaloneIT;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.file.metadata.StringArrayDeviceID;
+import org.apache.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.tsfile.read.TsFileSequenceReader;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.iotdb.db.it.utils.TestUtils.prepareTableData;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
+public class IoTDBCopyToTsFileIT {
+
+ private static final String DATABASE_NAME = "test_db";
+
+ protected static final String[] createSqls =
+ new String[] {
+ "CREATE DATABASE " + DATABASE_NAME,
+ "USE " + DATABASE_NAME,
+ "create table table1(tag1 string tag, tag2 string tag, s1 int32 field, s2 int32 field)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (1, 't1_1', 't2', 1, 1)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (2, 't1_1', 't2', 2, 2)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (3, 't1_1', 't2', 3, 3)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (1, 't1_2', 't2', 1, 1)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (2, 't1_2', 't2', 2, 2)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (3, 't1_2', 't2', 3, 3)",
+ "create table table2(tag1 string tag, tag2 string tag, s1 int32 field, s2 int32 field)",
+ "insert into table2(time, tag1, tag2, s1, s2) values (1, 't1_1', 't2', 1, 1)",
+ };
+
+ private String targetFilePath = null;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ EnvFactory.getEnv().initClusterEnvironment();
+ prepareTableData(createSqls);
+ }
+
+ @After
+ public void tearDownAfterTest() {
+ if (targetFilePath != null) {
+ try {
+ Files.deleteIfExists(new File(targetFilePath).toPath());
+ } catch (Exception ignored) {
+ }
+ targetFilePath = null;
+ }
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void testCopyTable()
+ throws IoTDBConnectionException, StatementExecutionException, IOException {
+ try (ITableSession session =
+ EnvFactory.getEnv().getTableSessionConnectionWithDB(DATABASE_NAME)) {
+ SessionDataSet sessionDataSet =
+ session.executeQueryStatement("copy table1 to '1.tsfile' (memory_threshold 1000000)");
+ SessionDataSet.DataIterator iterator = sessionDataSet.iterator();
+ while (iterator.next()) {
+ targetFilePath = iterator.getString(1);
+ int rowCount = iterator.getInt(2);
+ int deviceCount = iterator.getInt(3);
+ long sizeInBytes = iterator.getLong(4);
+ String tableName = iterator.getString(5);
+ String timeColumn = iterator.getString(6);
+ String tagColumns = iterator.getString(7);
+
+ Assert.assertTrue(new File(targetFilePath).exists());
+ Assert.assertEquals(6, rowCount);
+ Assert.assertEquals(2, deviceCount);
Review Comment:
COPY_TO_TSFILE_COLUMN_HEADERS defines row_count and device_count as INT64, but this test reads them via iterator.getInt(...), which truncates INT64 values and can silently overflow for large exports. Prefer using getLong(...) (and long variables) for these columns.
```suggestion
long rowCount = iterator.getLong(2);
long deviceCount = iterator.getLong(3);
long sizeInBytes = iterator.getLong(4);
String tableName = iterator.getString(5);
String timeColumn = iterator.getString(6);
String tagColumns = iterator.getString(7);
Assert.assertTrue(new File(targetFilePath).exists());
Assert.assertEquals(6L, rowCount);
Assert.assertEquals(2L, deviceCount);
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/node/CopyToNode.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.node;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.commons.schema.column.ColumnHeaderConstant;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.CopyToOptions;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanVisitor;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.process.SingleChildProcessNode;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.Symbol;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class CopyToNode extends SingleChildProcessNode {
+
+ private final String targetFilePath;
+ private final CopyToOptions copyToOptions;
+ private final List<Symbol> childPermittedOutputs;
+ private final List<Symbol> innerQueryOutputSymbols;
+ private final DatasetHeader innerQueryDatasetHeader;
+
+ public CopyToNode(
+ PlanNodeId id,
+ PlanNode child,
+ String targetFilePath,
+ CopyToOptions copyToOptions,
+ List<Symbol> childPermittedOutputs,
+ DatasetHeader innerQueryDatasetHeader,
+ List<Symbol> innerQueryOutputSymbols) {
+ super(id);
+ this.child = child;
+ this.targetFilePath = targetFilePath;
+ this.copyToOptions = copyToOptions;
+ this.childPermittedOutputs = childPermittedOutputs;
+ this.innerQueryDatasetHeader = innerQueryDatasetHeader;
+ this.innerQueryOutputSymbols = innerQueryOutputSymbols;
+ }
+
+ public DatasetHeader getInnerQueryDatasetHeader() {
+ return innerQueryDatasetHeader;
+ }
+
+ public List<Symbol> getInnerQueryOutputSymbols() {
+ return innerQueryOutputSymbols;
+ }
+
+ public String getTargetFilePath() {
+ return targetFilePath;
+ }
+
+ public CopyToOptions getCopyToOptions() {
+ return copyToOptions;
+ }
+
+ public List<Symbol> getChildPermittedOutputs() {
+ return childPermittedOutputs;
+ }
+
+ @Override
+ public PlanNode clone() {
+ return new CopyToNode(
+ id,
+ child,
+ targetFilePath,
+ copyToOptions,
+ childPermittedOutputs,
+ innerQueryDatasetHeader,
+ innerQueryOutputSymbols);
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitCopyTo(this, context);
+ }
+
+ @Override
+ public List<Symbol> getOutputSymbols() {
+ return ColumnHeaderConstant.COPY_TO_TSFILE_COLUMN_HEADERS.stream()
+ .map(column -> new Symbol(column.getColumnName()))
+ .collect(Collectors.toList());
+ }
+
+ @Override
+ public List<String> getOutputColumnNames() {
+ return ColumnHeaderConstant.COPY_TO_TSFILE_COLUMN_HEADERS.stream()
Review Comment:
CopyToNode.getOutputSymbols()/getOutputColumnNames() are hard-wired to ColumnHeaderConstant.COPY_TO_TSFILE_COLUMN_HEADERS even though the node already carries copyToOptions with getRespColumnHeaders(). Using the options here avoids duplicating the response schema logic and prevents mismatches if more COPY formats/options are added later.
```suggestion
return copyToOptions.getRespColumnHeaders().stream()
.map(column -> new Symbol(column.getColumnName()))
.collect(Collectors.toList());
}
@Override
public List<String> getOutputColumnNames() {
return copyToOptions.getRespColumnHeaders().stream()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4141517864
## [Codecov](https://app.codecov.io/gh/apache/iotdb/pull/17372?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `7.78689%` with `450 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 39.73%. Comparing base ([`402abdd`](https://app.codecov.io/gh/apache/iotdb/commit/402abdd00717590f943ef663117e6b8137d7c12e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`500131a`](https://app.codecov.io/gh/apache/iotdb/commit/500131a74c22906d6fe329ee6f478908ff13a27c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 20 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/iotdb/pull/17372?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...tor/process/copyto/tsfile/CopyToTsFileOptions.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fprocess%2Fcopyto%2Ftsfile%2FCopyToTsFileOptions.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3Byb2Nlc3MvY29weXRvL3RzZmlsZS9Db3B5VG9Uc0ZpbGVPcHRpb25zLmphdmE=) | 0.00% | [92 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rocess/copyto/tsfile/TsFileFormatCopyToWriter.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fprocess%2Fcopyto%2Ftsfile%2FTsFileFormatCopyToWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3Byb2Nlc3MvY29weXRvL3RzZmlsZS9Uc0ZpbGVGb3JtYXRDb3B5VG9Xcml0ZXIuamF2YQ==) | 0.00% | [88 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/operator/process/copyto/TableCopyToOperator.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fprocess%2Fcopyto%2FTableCopyToOperator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3Byb2Nlc3MvY29weXRvL1RhYmxlQ29weVRvT3BlcmF0b3IuamF2YQ==) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...yengine/plan/relational/sql/parser/AstBuilder.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fsql%2Fparser%2FAstBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3NxbC9wYXJzZXIvQXN0QnVpbGRlci5qYXZh) | 13.88% | [31 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...db/queryengine/plan/relational/sql/ast/CopyTo.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fsql%2Fast%2FCopyTo.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3NxbC9hc3QvQ29weVRvLmphdmE=) | 0.00% | [26 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/plan/relational/planner/TableLogicalPlanner.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fplanner%2FTableLogicalPlanner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3BsYW5uZXIvVGFibGVMb2dpY2FsUGxhbm5lci5qYXZh) | 32.43% | [25 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ngine/plan/relational/planner/node/CopyToNode.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fplanner%2Fnode%2FCopyToNode.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3BsYW5uZXIvbm9kZS9Db3B5VG9Ob2RlLmphdmE=) | 0.00% | [25 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ecution/operator/process/copyto/CopyToOptions.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fexecution%2Foperator%2Fprocess%2Fcopyto%2FCopyToOptions.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvZXhlY3V0aW9uL29wZXJhdG9yL3Byb2Nlc3MvY29weXRvL0NvcHlUb09wdGlvbnMuamF2YQ==) | 0.00% | [23 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...eryengine/plan/planner/TableOperatorGenerator.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2FTableOperatorGenerator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL1RhYmxlT3BlcmF0b3JHZW5lcmF0b3IuamF2YQ==) | 0.00% | [15 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...elational/planner/distribute/AddExchangeNodes.java](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&filepath=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Frelational%2Fplanner%2Fdistribute%2FAddExchangeNodes.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9yZWxhdGlvbmFsL3BsYW5uZXIvZGlzdHJpYnV0ZS9BZGRFeGNoYW5nZU5vZGVzLmphdmE=) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [11 more](https://app.codecov.io/gh/apache/iotdb/pull/17372?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17372 +/- ##
============================================
- Coverage 39.74% 39.73% -0.02%
Complexity 312 312
============================================
Files 5126 5134 +8
Lines 346228 346796 +568
Branches 44083 44172 +89
============================================
+ Hits 137608 137798 +190
- Misses 208620 208998 +378
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/iotdb/pull/17372?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4153543726
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate failed**
Failed conditions
 [B Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) (required ≥ A)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 merged PR #17372:
URL: https://github.com/apache/iotdb/pull/17372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4151737579
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [24 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4140941594
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [24 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4141173901
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [22 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4151663783
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [24 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "JackieTien97 (via GitHub)" <gi...@apache.org>.
JackieTien97 commented on code in PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#discussion_r3007363397
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/copyto/IoTDBCopyToTsFileIT.java:
##########
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.relational.it.query.recent.copyto;
+
+import org.apache.iotdb.isession.ITableSession;
+import org.apache.iotdb.isession.SessionDataSet;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.TableClusterIT;
+import org.apache.iotdb.itbase.category.TableLocalStandaloneIT;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.file.metadata.StringArrayDeviceID;
+import org.apache.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.tsfile.read.TsFileSequenceReader;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.iotdb.db.it.utils.TestUtils.prepareTableData;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
+public class IoTDBCopyToTsFileIT {
+
+ private static final String DATABASE_NAME = "test_db";
+
+ protected static final String[] createSqls =
+ new String[] {
+ "CREATE DATABASE " + DATABASE_NAME,
+ "USE " + DATABASE_NAME,
+ "create table table1(tag1 string tag, tag2 string tag, s1 int32 field, s2 int32 field)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (1, 't1_1', 't2', 1, 1)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (2, 't1_1', 't2', 2, 2)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (3, 't1_1', 't2', 3, 3)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (1, 't1_2', 't2', 1, 1)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (2, 't1_2', 't2', 2, 2)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (3, 't1_2', 't2', 3, 3)",
+ "create table table2(tag1 string tag, tag2 string tag, s1 int32 field, s2 int32 field)",
+ "insert into table2(time, tag1, tag2, s1, s2) values (1, 't1_1', 't2', 1, 1)",
+ };
+
+ private String targetFilePath = null;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ EnvFactory.getEnv().initClusterEnvironment();
+ prepareTableData(createSqls);
+ }
+
+ @After
+ public void tearDownAfterTest() {
+ if (targetFilePath != null) {
+ try {
+ Files.deleteIfExists(new File(targetFilePath).toPath());
+ } catch (Exception ignored) {
+ }
+ targetFilePath = null;
+ }
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void testCopyTable()
+ throws IoTDBConnectionException, StatementExecutionException, IOException {
+ try (ITableSession session =
+ EnvFactory.getEnv().getTableSessionConnectionWithDB(DATABASE_NAME)) {
+ SessionDataSet sessionDataSet =
+ session.executeQueryStatement("copy table1 to '1.tsfile' (memory_threshold 1000000)");
+ SessionDataSet.DataIterator iterator = sessionDataSet.iterator();
+ while (iterator.next()) {
+ targetFilePath = iterator.getString(1);
+ int rowCount = iterator.getInt(2);
+ int deviceCount = iterator.getInt(3);
Review Comment:
should get long?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableModelTypeProviderExtractor.java:
##########
@@ -101,6 +102,12 @@ private void addOutputSymbolsToTypeProvider(PlanNode node) {
}
}
+ @Override
+ public Void visitCopyTo(CopyToNode node, Void context) {
+ visitPlan(node.getChild(), context);
+ return null;
Review Comment:
add output symbol type to beTypeProvider.putTableModelType, just like intonode?
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/node/CopyToNode.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.node;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.commons.schema.column.ColumnHeaderConstant;
+import org.apache.iotdb.db.queryengine.common.header.DatasetHeader;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.CopyToOptions;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanNodeId;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.PlanVisitor;
+import org.apache.iotdb.db.queryengine.plan.planner.plan.node.process.SingleChildProcessNode;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.Symbol;
+
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class CopyToNode extends SingleChildProcessNode {
+
+ private final String targetFilePath;
+ private final CopyToOptions copyToOptions;
+ private final List<Symbol> childPermittedOutputs;
+ private final List<Symbol> innerQueryOutputSymbols;
+ private final DatasetHeader innerQueryDatasetHeader;
+
+ public CopyToNode(
+ PlanNodeId id,
+ PlanNode child,
+ String targetFilePath,
+ CopyToOptions copyToOptions,
+ List<Symbol> childPermittedOutputs,
+ DatasetHeader innerQueryDatasetHeader,
+ List<Symbol> innerQueryOutputSymbols) {
+ super(id);
+ this.child = child;
+ this.targetFilePath = targetFilePath;
+ this.copyToOptions = copyToOptions;
+ this.childPermittedOutputs = childPermittedOutputs;
+ this.innerQueryDatasetHeader = innerQueryDatasetHeader;
+ this.innerQueryOutputSymbols = innerQueryOutputSymbols;
+ }
+
+ public DatasetHeader getInnerQueryDatasetHeader() {
+ return innerQueryDatasetHeader;
+ }
+
+ public List<Symbol> getInnerQueryOutputSymbols() {
+ return innerQueryOutputSymbols;
+ }
+
+ public String getTargetFilePath() {
+ return targetFilePath;
+ }
+
+ public CopyToOptions getCopyToOptions() {
+ return copyToOptions;
+ }
+
+ public List<Symbol> getChildPermittedOutputs() {
+ return childPermittedOutputs;
+ }
+
+ @Override
+ public PlanNode clone() {
+ return new CopyToNode(
+ id,
+ child,
+ targetFilePath,
+ copyToOptions,
+ childPermittedOutputs,
+ innerQueryDatasetHeader,
+ innerQueryOutputSymbols);
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitCopyTo(this, context);
+ }
+
+ @Override
+ public List<Symbol> getOutputSymbols() {
Review Comment:
This recreates Symbol objects on every invocation. Since getOutputSymbols() is called frequently during planning, this is wasteful. Cache the result or use a static final list.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/tsfile/TsFileFormatCopyToWriter.java:
##########
@@ -0,0 +1,196 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.commons.schema.column.ColumnHeaderConstant;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.IFormatCopyToWriter;
+
+import org.apache.ratis.util.MemoizedCheckedSupplier;
+import org.apache.tsfile.common.conf.TSFileConfig;
+import org.apache.tsfile.enums.ColumnCategory;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.ColumnSchema;
+import org.apache.tsfile.file.metadata.TableSchema;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.read.common.block.TsBlockBuilder;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.apache.tsfile.write.v4.TableTsBlock2TsFileWriter;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class TsFileFormatCopyToWriter implements IFormatCopyToWriter {
+ private final File targetFile;
+ private final String targetTableName;
+ private final String targetTimeColumn;
+ private final Set<String> targetTagColumns;
+ private final boolean generateNewTimeColumn;
+
+ private MemoizedCheckedSupplier<TableTsBlock2TsFileWriter, IOException> tsFileWriter;
+ private long rowCount = 0;
+ private long deviceCount = 0;
+
+ public TsFileFormatCopyToWriter(
+ File file,
+ CopyToTsFileOptions copyToOptions,
+ List<ColumnHeader> innerQueryDatasetHeader,
+ int[] columnIndex2TsBlockColumnIndex) {
+ this.targetFile = file;
+ this.targetTableName = copyToOptions.getTargetTableName();
+ this.targetTimeColumn = copyToOptions.getTargetTimeColumn();
+ this.generateNewTimeColumn = copyToOptions.isGenerateNewTimeColumn();
+ targetTagColumns = copyToOptions.getTargetTagColumns();
+
+ ColumnHeader[] columnHeadersMatchChildTsBlock =
+ getColumnHeadersMatchTsBlock(innerQueryDatasetHeader, columnIndex2TsBlockColumnIndex);
+ List<ColumnSchema> columnSchemas =
+ new ArrayList<>(columnHeadersMatchChildTsBlock.length + (generateNewTimeColumn ? 1 : 0));
+ Map<String, Integer> columnNameIndexMapInDatasetHeader = new HashMap<>();
+ for (int i = 0; i < columnHeadersMatchChildTsBlock.length; i++) {
+ columnNameIndexMapInDatasetHeader.put(columnHeadersMatchChildTsBlock[i].getColumnName(), i);
+ }
+ // add time column
+ columnSchemas.add(
+ new ColumnSchema(targetTimeColumn, TSDataType.TIMESTAMP, ColumnCategory.TIME));
+ int timeColumnIdxInQueryTsBlock =
+ generateNewTimeColumn ? -1 : columnNameIndexMapInDatasetHeader.get(targetTimeColumn);
+
+ // add tag columns
+ int[] tagColumnIndexesInTsBlock = new int[targetTagColumns.size()];
+ int arrIdx = 0;
+ for (String targetTagColumn : targetTagColumns) {
+ int tagIdx = columnNameIndexMapInDatasetHeader.get(targetTagColumn);
+ ColumnHeader tagColumnHeader = columnHeadersMatchChildTsBlock[tagIdx];
+ columnSchemas.add(
+ new ColumnSchema(
+ tagColumnHeader.getColumnName(),
+ tagColumnHeader.getColumnType(),
+ ColumnCategory.TAG));
+ tagColumnIndexesInTsBlock[arrIdx++] = tagIdx;
+ }
+ // add field columns
+ int fieldColumnCount =
+ columnHeadersMatchChildTsBlock.length
+ - tagColumnIndexesInTsBlock.length
+ - (generateNewTimeColumn ? 0 : 1);
+ int[] fieldColumnIndexesInQueryTsBlock = new int[fieldColumnCount];
+ IMeasurementSchema[] fieldColumnSchemas = new IMeasurementSchema[fieldColumnCount];
+ arrIdx = 0;
+ for (int i = 0; i < columnHeadersMatchChildTsBlock.length; i++) {
+ ColumnHeader columnHeader = columnHeadersMatchChildTsBlock[i];
+ if (targetTagColumns.contains(columnHeader.getColumnName())
+ || columnHeader.getColumnName().equals(targetTimeColumn)) {
+ continue;
+ }
+ columnSchemas.add(
+ new ColumnSchema(
+ columnHeader.getColumnName(), columnHeader.getColumnType(), ColumnCategory.FIELD));
+ fieldColumnSchemas[arrIdx] =
+ new MeasurementSchema(columnHeader.getColumnName(), columnHeader.getColumnType());
+ fieldColumnIndexesInQueryTsBlock[arrIdx] = i;
+ arrIdx++;
+ }
+
+ TableSchema tableSchema = new TableSchema(targetTableName, columnSchemas);
+ this.tsFileWriter =
+ MemoizedCheckedSupplier.valueOf(
+ () ->
+ new TableTsBlock2TsFileWriter(
+ targetFile,
+ tableSchema,
+ copyToOptions.getTargetMemoryThreshold(),
+ generateNewTimeColumn,
+ timeColumnIdxInQueryTsBlock,
+ tagColumnIndexesInTsBlock,
+ fieldColumnIndexesInQueryTsBlock,
+ fieldColumnSchemas));
+ }
+
+ private ColumnHeader[] getColumnHeadersMatchTsBlock(
+ List<ColumnHeader> queryOutputColumnHeaders, int[] columnIndex2TsBlockColumnIndexList) {
+ ColumnHeader[] columnHeadersMatchTsBlock = new ColumnHeader[queryOutputColumnHeaders.size()];
+ for (int i = 0; i < columnIndex2TsBlockColumnIndexList.length; i++) {
+ int tsBlockIndex = columnIndex2TsBlockColumnIndexList[i];
+ columnHeadersMatchTsBlock[tsBlockIndex] = queryOutputColumnHeaders.get(i);
+ }
+ return columnHeadersMatchTsBlock;
+ }
+
+ @Override
+ public void write(TsBlock tsBlock) throws Exception {
+ tsFileWriter.get().write(tsBlock);
+ }
+
+ @Override
+ public void seal() throws Exception {
+ if (!tsFileWriter.isInitialized()) {
+ return;
+ }
+ TableTsBlock2TsFileWriter writer = tsFileWriter.get();
+ writer.close();
+ // should call these methods after writer.close()
+ deviceCount = writer.getDeviceCount();
+ rowCount = writer.getRowCount();
+ tsFileWriter = null;
+ }
+
+ @Override
+ public TsBlock buildResultTsBlock() {
+ TsBlockBuilder builder =
+ TsBlockBuilder.withMaxTsBlockSize(
+ 1024,
+ ColumnHeaderConstant.COPY_TO_TSFILE_COLUMN_HEADERS.stream()
+ .map(ColumnHeader::getColumnType)
+ .collect(Collectors.toList()));
+ builder.getTimeColumnBuilder().writeLong(0);
+ builder.getValueColumnBuilders()[0].writeBinary(
+ new Binary(rowCount > 0 ? targetFile.getAbsolutePath() : "", TSFileConfig.STRING_CHARSET));
+ builder.getValueColumnBuilders()[1].writeLong(rowCount);
+ builder.getValueColumnBuilders()[2].writeLong(deviceCount);
+ builder.getValueColumnBuilders()[3].writeLong(targetFile.length());
+ builder.getValueColumnBuilders()[4].writeBinary(
+ new Binary(targetTableName, TSFileConfig.STRING_CHARSET));
+ builder.getValueColumnBuilders()[5].writeBinary(
+ new Binary(
+ targetTimeColumn + (generateNewTimeColumn ? "(auto_gen)" : ""),
Review Comment:
make "(auto_gen)" a constant
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/TableCopyToOperator.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.CopyToTsFileOptions;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.TsFileFormatCopyToWriter;
+import org.apache.iotdb.db.storageengine.rescon.disk.TierManager;
+
+import com.google.common.util.concurrent.ListenableFuture;
+import org.apache.tsfile.common.conf.TSFileDescriptor;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+
+public class TableCopyToOperator implements ProcessOperator {
+ private static final long INSTANCE_SIZE =
+ RamUsageEstimator.shallowSizeOfInstance(TableCopyToOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator childOperator;
+ private final String targetFilePath;
+ private final CopyToOptions options;
+ private final List<ColumnHeader> innerQueryColumnHeaders;
+ private final int[] columnIndex2TsBlockColumnIndex;
+
+ private IFormatCopyToWriter writer;
+ private File targetFile;
+ private boolean isFinished = false;
+ private boolean hasData = false;
+
+ public TableCopyToOperator(
+ OperatorContext operatorContext,
+ Operator child,
+ String targetFilePath,
+ CopyToOptions options,
+ List<ColumnHeader> innerQueryColumnHeaders,
+ int[] columnIndex2TsBlockColumnIndex) {
+ this.operatorContext = operatorContext;
+ this.childOperator = child;
+ this.targetFilePath = targetFilePath;
+ this.options = options;
+ this.innerQueryColumnHeaders = innerQueryColumnHeaders;
+ this.columnIndex2TsBlockColumnIndex = columnIndex2TsBlockColumnIndex;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+ return operatorContext;
+ }
+
+ @Override
+ public TsBlock next() throws Exception {
+ IFormatCopyToWriter formatWriter = getWriter();
+ if (!childOperator.hasNext()) {
+ isFinished = true;
+ formatWriter.seal();
+ return formatWriter.buildResultTsBlock();
+ }
+ TsBlock tsBlock = childOperator.next();
+ if (tsBlock == null || tsBlock.isEmpty()) {
+ return null;
+ }
+ hasData = true;
+ formatWriter.write(tsBlock);
+ return null;
+ }
+
+ private IFormatCopyToWriter getWriter() throws Exception {
+ if (writer != null) {
+ return writer;
+ }
+ this.targetFile = createTargetFile(targetFilePath);
+ switch (options.getFormat()) {
+ case TSFILE:
+ default:
+ this.writer =
+ new TsFileFormatCopyToWriter(
+ this.targetFile,
+ (CopyToTsFileOptions) options,
+ innerQueryColumnHeaders,
+ columnIndex2TsBlockColumnIndex);
+ }
+ return writer;
+ }
+
+ private File createTargetFile(String path) throws Exception {
+ File file = new File(path);
+ if (file.getParent() == null) {
+ String dir = TierManager.getInstance().getNextFolderForCopyToTargetFile();
+ file = new File(dir, path);
+ }
+ File parent = file.getParentFile();
+ if (parent != null && !parent.exists() && !parent.mkdirs()) {
+ throw new IOException("Failed to create directories: " + parent);
+ }
+ if (file.exists()) {
+ throw new IOException("Target file already exists: " + file.getAbsolutePath());
+ }
+ if (!file.createNewFile()) {
+ throw new IOException("Failed to create file: " + file.getAbsolutePath());
+ }
+
+ return file;
+ }
+
+ @Override
+ public boolean hasNext() throws Exception {
+ return !isFinished;
+ }
+
+ @Override
+ public ListenableFuture<?> isBlocked() {
+ return childOperator.isBlocked();
+ }
+
+ @Override
+ public void close() throws Exception {
+ childOperator.close();
+ if (writer != null) {
+ writer.close();
+ writer = null;
+ }
+ if (targetFile == null || (hasData && isFinished)) {
+ return;
+ }
+ Files.deleteIfExists(targetFile.toPath());
+ }
+
+ @Override
+ public boolean isFinished() throws Exception {
+ return isFinished;
+ }
+
+ @Override
+ public long calculateMaxPeekMemory() {
Review Comment:
should include the memory occupied by tsfile writer, that's the memoryThreshold
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/TableCopyToOperator.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.CopyToTsFileOptions;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.TsFileFormatCopyToWriter;
+import org.apache.iotdb.db.storageengine.rescon.disk.TierManager;
+
+import com.google.common.util.concurrent.ListenableFuture;
+import org.apache.tsfile.common.conf.TSFileDescriptor;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+
+public class TableCopyToOperator implements ProcessOperator {
+ private static final long INSTANCE_SIZE =
+ RamUsageEstimator.shallowSizeOfInstance(TableCopyToOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator childOperator;
+ private final String targetFilePath;
+ private final CopyToOptions options;
+ private final List<ColumnHeader> innerQueryColumnHeaders;
+ private final int[] columnIndex2TsBlockColumnIndex;
+
+ private IFormatCopyToWriter writer;
+ private File targetFile;
+ private boolean isFinished = false;
+ private boolean hasData = false;
+
+ public TableCopyToOperator(
+ OperatorContext operatorContext,
+ Operator child,
+ String targetFilePath,
+ CopyToOptions options,
+ List<ColumnHeader> innerQueryColumnHeaders,
+ int[] columnIndex2TsBlockColumnIndex) {
+ this.operatorContext = operatorContext;
+ this.childOperator = child;
+ this.targetFilePath = targetFilePath;
+ this.options = options;
+ this.innerQueryColumnHeaders = innerQueryColumnHeaders;
+ this.columnIndex2TsBlockColumnIndex = columnIndex2TsBlockColumnIndex;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+ return operatorContext;
+ }
+
+ @Override
+ public TsBlock next() throws Exception {
+ IFormatCopyToWriter formatWriter = getWriter();
+ if (!childOperator.hasNext()) {
+ isFinished = true;
+ formatWriter.seal();
Review Comment:
if seal() throws, isFinished is set to true on line 81 before seal() on line 82. So isFinished = true and hasData = true→ the file won't be cleaned up even though sealing failed.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/CopyToOptions.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.db.exception.sql.SemanticException;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.CopyToTsFileOptions;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlan;
+
+import org.apache.tsfile.utils.Accountable;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * Interface for COPY TO command options.
+ *
+ * <p>This interface defines the configuration for COPY TO operations, including target format,
+ * target table name, column mappings, and memory management. Implementations provide
+ * format-specific validation and inference logic.
+ */
+public interface CopyToOptions extends Accountable {
+
+ /**
+ * Infers and validates options based on query analysis and relation plan.
+ *
+ * <p>This method is called during analysis phase to fill in default values and validate
+ * user-specified options against the query structure. For example, it can infer the target time
+ * column from the query's time column if not explicitly specified.
+ *
+ * @param analysis the query analysis containing metadata about the query
+ * @param queryRelationPlan the logical relation plan of the inner query
+ * @param columnHeaders the column headers from the query result
+ */
+ void infer(Analysis analysis, RelationPlan queryRelationPlan, List<ColumnHeader> columnHeaders);
+
+ /**
+ * Validates the options against the actual column schema.
+ *
+ * <p>This method is called after planning to ensure the specified options (e.g., target columns,
+ * tags) are valid for the given output schema.
+ *
+ * @param columnHeaders the column headers of the query result
+ * @throws SemanticException if validation fails
+ */
+ void check(List<ColumnHeader> columnHeaders);
+
+ /**
+ * Returns the response column headers for the result set.
+ *
+ * <p>These headers describe the columns that will be returned to the client after the COPY TO
+ * operation completes (e.g., file path, row count).
+ *
+ * @return list of column headers for the response
+ */
+ List<ColumnHeader> getRespColumnHeaders();
+
+ /**
+ * Returns the target output format.
+ *
+ * @return the format enum value
+ */
+ Format getFormat();
+
+ /**
+ * Estimates the maximum memory usage in bytes required for writing.
+ *
+ * <p>This is used for memory management and determining whether to flush data to disk.
+ *
+ * @return estimated maximum memory usage in bytes
+ */
+ long estimatedMaxRamBytesInWrite();
+
+ /** Supported output formats for COPY TO command. */
+ enum Format {
+ /** TsFile format output. */
+ TSFILE,
+ }
+
+ class Builder {
+ private CopyToOptions.Format format = CopyToOptions.Format.TSFILE;
+ private String targetTableName = null;
+ private String targetTimeColumn = null;
+ private Set<String> targetTagColumns = null;
+ private long memoryThreshold = 0;
+
+ public Builder withFormat(CopyToOptions.Format format) {
+ this.format = format;
+ return this;
+ }
+
+ public Builder withTargetTableName(String targetTableName) {
+ this.targetTableName = targetTableName;
+ return this;
+ }
+
+ public Builder withTargetTimeColumn(String targetTimeColumn) {
+ this.targetTimeColumn = targetTimeColumn;
+ return this;
+ }
+
+ public Builder withTargetTagColumns(Set<String> targetTagColumns) {
+ this.targetTagColumns = targetTagColumns;
+ return this;
+ }
+
+ public Builder withMemoryThreshold(long memoryThreshold) {
+ if (memoryThreshold <= 0) {
+ throw new SemanticException("The memory threshold must be greater than 0.");
+ }
+ this.memoryThreshold = memoryThreshold;
+ return this;
+ }
+
+ public CopyToOptions build() {
+ switch (format) {
+ case TSFILE:
+ default:
+ return new CopyToTsFileOptions(
+ targetTableName,
+ targetTimeColumn,
+ targetTagColumns,
+ memoryThreshold != 0 ? memoryThreshold : 32 * 1024 * 1024);
Review Comment:
keep it to `0`, means using the default value defined in tsfile writer api, no need to using `32 * 1024 * 1024` magic number
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/query/recent/copyto/IoTDBCopyToTsFileIT.java:
##########
@@ -0,0 +1,490 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.relational.it.query.recent.copyto;
+
+import org.apache.iotdb.isession.ITableSession;
+import org.apache.iotdb.isession.SessionDataSet;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.TableClusterIT;
+import org.apache.iotdb.itbase.category.TableLocalStandaloneIT;
+import org.apache.iotdb.rpc.IoTDBConnectionException;
+import org.apache.iotdb.rpc.StatementExecutionException;
+
+import org.apache.tsfile.file.metadata.IDeviceID;
+import org.apache.tsfile.file.metadata.StringArrayDeviceID;
+import org.apache.tsfile.file.metadata.TimeseriesMetadata;
+import org.apache.tsfile.read.TsFileSequenceReader;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.iotdb.db.it.utils.TestUtils.prepareTableData;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
+public class IoTDBCopyToTsFileIT {
+
+ private static final String DATABASE_NAME = "test_db";
+
+ protected static final String[] createSqls =
+ new String[] {
+ "CREATE DATABASE " + DATABASE_NAME,
+ "USE " + DATABASE_NAME,
+ "create table table1(tag1 string tag, tag2 string tag, s1 int32 field, s2 int32 field)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (1, 't1_1', 't2', 1, 1)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (2, 't1_1', 't2', 2, 2)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (3, 't1_1', 't2', 3, 3)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (1, 't1_2', 't2', 1, 1)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (2, 't1_2', 't2', 2, 2)",
+ "insert into table1(time, tag1, tag2, s1, s2) values (3, 't1_2', 't2', 3, 3)",
+ "create table table2(tag1 string tag, tag2 string tag, s1 int32 field, s2 int32 field)",
+ "insert into table2(time, tag1, tag2, s1, s2) values (1, 't1_1', 't2', 1, 1)",
+ };
+
+ private String targetFilePath = null;
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ EnvFactory.getEnv().initClusterEnvironment();
+ prepareTableData(createSqls);
+ }
+
+ @After
+ public void tearDownAfterTest() {
+ if (targetFilePath != null) {
+ try {
+ Files.deleteIfExists(new File(targetFilePath).toPath());
+ } catch (Exception ignored) {
Review Comment:
While acceptable in test cleanup, this could silently mask test infrastructure issues. Consider logging a warning.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/disk/TierManager.java:
##########
@@ -226,6 +247,10 @@ public String getNextFolderForTsFile(int tierLevel, boolean sequence)
: unSeqTiers.get(tierLevel).getNextFolder();
}
+ public String getNextFolderForCopyToTargetFile() throws DiskSpaceInsufficientException {
Review Comment:
@shuwenwei why marking it as resolved, it seems that it's not resolved yet.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/TableCopyToOperator.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.CopyToTsFileOptions;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.TsFileFormatCopyToWriter;
+import org.apache.iotdb.db.storageengine.rescon.disk.TierManager;
+
+import com.google.common.util.concurrent.ListenableFuture;
+import org.apache.tsfile.common.conf.TSFileDescriptor;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+
+public class TableCopyToOperator implements ProcessOperator {
+ private static final long INSTANCE_SIZE =
+ RamUsageEstimator.shallowSizeOfInstance(TableCopyToOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator childOperator;
+ private final String targetFilePath;
+ private final CopyToOptions options;
+ private final List<ColumnHeader> innerQueryColumnHeaders;
+ private final int[] columnIndex2TsBlockColumnIndex;
+
+ private IFormatCopyToWriter writer;
+ private File targetFile;
+ private boolean isFinished = false;
+ private boolean hasData = false;
+
+ public TableCopyToOperator(
+ OperatorContext operatorContext,
+ Operator child,
+ String targetFilePath,
+ CopyToOptions options,
+ List<ColumnHeader> innerQueryColumnHeaders,
+ int[] columnIndex2TsBlockColumnIndex) {
+ this.operatorContext = operatorContext;
+ this.childOperator = child;
+ this.targetFilePath = targetFilePath;
+ this.options = options;
+ this.innerQueryColumnHeaders = innerQueryColumnHeaders;
+ this.columnIndex2TsBlockColumnIndex = columnIndex2TsBlockColumnIndex;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+ return operatorContext;
+ }
+
+ @Override
+ public TsBlock next() throws Exception {
+ IFormatCopyToWriter formatWriter = getWriter();
+ if (!childOperator.hasNext()) {
+ isFinished = true;
+ formatWriter.seal();
Review Comment:
```suggestion
formatWriter.seal();
isFinished = true;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#discussion_r2999539201
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/TableCopyToOperator.java:
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.db.queryengine.execution.MemoryEstimationHelper;
+import org.apache.iotdb.db.queryengine.execution.operator.Operator;
+import org.apache.iotdb.db.queryengine.execution.operator.OperatorContext;
+import org.apache.iotdb.db.queryengine.execution.operator.process.ProcessOperator;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.CopyToTsFileOptions;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile.TsFileFormatCopyToWriter;
+import org.apache.iotdb.db.storageengine.rescon.disk.TierManager;
+
+import com.google.common.util.concurrent.ListenableFuture;
+import org.apache.tsfile.common.conf.TSFileDescriptor;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+public class TableCopyToOperator implements ProcessOperator {
+ private static final long INSTANCE_SIZE =
+ RamUsageEstimator.shallowSizeOfInstance(TableCopyToOperator.class);
+
+ private final OperatorContext operatorContext;
+ private final Operator childOperator;
+ private final String targetFilePath;
+ private final CopyToOptions options;
+ private final List<ColumnHeader> innerQueryColumnHeaders;
+ private final List<Integer> columnIndex2TsBlockColumnIndexList;
+
+ private IFormatCopyToWriter writer;
+ private File targetFile;
+ private boolean isFinished = false;
+ private boolean hasData = false;
+
+ public TableCopyToOperator(
+ OperatorContext operatorContext,
+ Operator child,
+ String targetFilePath,
+ CopyToOptions options,
+ List<ColumnHeader> innerQueryColumnHeaders,
+ List<Integer> columnIndex2TsBlockColumnIndexList) {
+ this.operatorContext = operatorContext;
+ this.childOperator = child;
+ this.targetFilePath = targetFilePath;
+ this.options = options;
+ this.innerQueryColumnHeaders = innerQueryColumnHeaders;
+ this.columnIndex2TsBlockColumnIndexList = columnIndex2TsBlockColumnIndexList;
+ }
+
+ @Override
+ public OperatorContext getOperatorContext() {
+ return operatorContext;
+ }
+
+ @Override
+ public TsBlock next() throws Exception {
+ long startTime = System.nanoTime();
+ long maxRuntime = operatorContext.getMaxRunTime().roundTo(TimeUnit.NANOSECONDS);
+ IFormatCopyToWriter formatWriter = getWriter();
+ do {
+ if (!childOperator.hasNext()) {
+ isFinished = true;
+ break;
+ }
+ TsBlock tsBlock = childOperator.next();
+ if (tsBlock == null || tsBlock.isEmpty()) {
+ continue;
+ }
+ hasData = true;
+ formatWriter.write(tsBlock);
+ } while (System.nanoTime() - startTime < maxRuntime && !isFinished);
+
+ if (isFinished) {
+ formatWriter.seal();
+ return formatWriter.buildResultTsBlock();
+ }
+ return null;
+ }
+
+ private IFormatCopyToWriter getWriter() throws Exception {
+ if (writer != null) {
+ return writer;
+ }
+ this.targetFile = createTargetFile(targetFilePath);
+ switch (options.getFormat()) {
+ case TSFILE:
+ default:
+ this.writer =
+ new TsFileFormatCopyToWriter(
+ this.targetFile,
+ (CopyToTsFileOptions) options,
+ innerQueryColumnHeaders,
+ columnIndex2TsBlockColumnIndexList);
+ }
+ return writer;
+ }
+
+ private File createTargetFile(String path) throws Exception {
+ File file = new File(path);
+ if (file.getParent() == null) {
+ String dir = TierManager.getInstance().getNextFolderForCopyToTargetFile();
+ file = new File(dir, path);
+ }
+ File parent = file.getParentFile();
+ if (parent != null && !parent.exists() && !parent.mkdirs()) {
+ throw new IOException("Failed to create directories: " + parent);
+ }
+ if (!file.createNewFile()) {
+ throw new IOException("Target file already exists: " + file.getAbsolutePath());
+ }
Review Comment:
`createTargetFile()` will create a new file at an arbitrary user-supplied path (including absolute paths and paths with `..`) and will also create parent directories. Since the path comes from a SQL statement, this effectively allows remote clients to write files anywhere the server user has permissions. Please restrict COPY TO targets to the dedicated `copy_to` directory (or another explicit allowlist/configured base dir), and reject absolute paths / path traversal.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/disk/TierManager.java:
##########
@@ -226,6 +247,10 @@ public String getNextFolderForTsFile(int tierLevel, boolean sequence)
: unSeqTiers.get(tierLevel).getNextFolder();
}
+ public String getNextFolderForCopyToTargetFile() throws DiskSpaceInsufficientException {
Review Comment:
`getNextFolderForCopyToTargetFile()` will throw a NullPointerException if `copyToFolderManager` was not initialized (e.g., `FolderManager` construction failed due to `DiskSpaceInsufficientException`). Consider either not swallowing the exception in `initFolders()`, or check for null here and throw a clear `DiskSpaceInsufficientException`/`IllegalStateException` with actionable context.
```suggestion
public String getNextFolderForCopyToTargetFile() throws DiskSpaceInsufficientException {
if (copyToFolderManager == null) {
throw new DiskSpaceInsufficientException(
"copyToFolderManager is not initialized. This usually indicates that folder "
+ "initialization in TierManager.initFolders() failed due to insufficient disk "
+ "space. Please check disk space and related configuration before retrying the "
+ "copy-to-target operation.");
}
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/optimizations/UnaliasSymbolReferences.java:
##########
@@ -388,6 +389,24 @@ public PlanAndMappings visitExplainAnalyze(ExplainAnalyzeNode node, UnaliasConte
mapping);
}
+ @Override
+ public PlanAndMappings visitCopyTo(CopyToNode node, UnaliasContext context) {
+ PlanAndMappings rewrittenSource = node.getChild().accept(this, context);
+ Map<Symbol, Symbol> mapping = new HashMap<>(rewrittenSource.getMappings());
+ SymbolMapper mapper = symbolMapper(mapping);
+ List<Symbol> newChildPermittedOutputs = mapper.map(node.getChildPermittedOutputs());
+ return new PlanAndMappings(
+ new CopyToNode(
+ node.getPlanNodeId(),
+ rewrittenSource.getRoot(),
+ node.getTargetFilePath(),
+ node.getCopyToOptions(),
+ newChildPermittedOutputs,
+ node.getInnerQueryDatasetHeader(),
+ node.getInnerQueryOutputNode()),
Review Comment:
In `visitCopyTo`, the child plan is rewritten and symbols may be remapped, but `innerQueryOutputNode` is passed through unchanged. `TableCopyToOperator` later calls `datasetHeader.setTableColumnToTsBlockIndexMap(innerQueryOutputNode, childNode)` and relies on `innerQueryOutputNode.getOutputSymbols()` matching the (possibly rewritten) child’s output symbols; if they don’t, `outputSymbolsIndexMap.get(...)` can return null and fail at runtime. Please remap `innerQueryOutputNode`’s output symbols using the same `SymbolMapper` (similar to `childPermittedOutputs`), or store a symbol-independent mapping instead of the original `OutputNode`.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/ast/CopyTo.java:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.sql.ast;
+
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.CopyToOptions;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.tsfile.utils.RamUsageEstimator;
+
+import javax.annotation.Nullable;
+
+import java.util.List;
+import java.util.Objects;
+
+public class CopyTo extends Statement {
+
+ private static final long INSTANCE_SIZE = RamUsageEstimator.shallowSizeOfInstance(CopyTo.class);
+ private final Statement queryStatement;
+ private final String targetFileName;
+ private final CopyToOptions options;
+
+ public CopyTo(Statement queryStatement, String targetFileName, CopyToOptions options) {
+ this(null, queryStatement, targetFileName, options);
+ }
+
+ public CopyTo(
+ @Nullable NodeLocation location,
+ Statement queryStatement,
+ String targetFileName,
+ CopyToOptions options) {
+ super(location);
+ this.queryStatement = queryStatement;
+ this.targetFileName = targetFileName;
+ this.options = options;
+ }
+
+ public Statement getQueryStatement() {
+ return queryStatement;
+ }
+
+ public String getTargetFileName() {
+ return targetFileName;
+ }
+
+ public CopyToOptions getOptions() {
+ return options;
+ }
+
+ @Override
+ public <R, C> R accept(AstVisitor<R, C> visitor, C context) {
+ return visitor.visitCopyTo(this, context);
+ }
+
+ @Override
+ public List<Node> getChildren() {
+ return ImmutableList.<Node>builder().add(queryStatement).build();
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (o == null || getClass() != o.getClass()) return false;
+ CopyTo copyTo = (CopyTo) o;
+ return Objects.equals(queryStatement, copyTo.queryStatement)
+ && Objects.equals(targetFileName, copyTo.targetFileName)
+ && Objects.equals(options, copyTo.options);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(queryStatement, targetFileName, options);
+ }
+
+ @Override
+ public String toString() {
+ return "CopyTo{"
+ + "queryStatement="
+ + queryStatement
+ + ", targetFileName='"
+ + targetFileName
+ + '\''
+ + ", options="
+ + options
+ + '}';
+ }
+
+ @Override
+ public long ramBytesUsed() {
+ long size = INSTANCE_SIZE;
+ size += AstMemoryEstimationHelper.getEstimatedSizeOfNodeLocation(getLocationInternal());
+ size += AstMemoryEstimationHelper.getEstimatedSizeOfAccountableObject(queryStatement);
Review Comment:
`ramBytesUsed()` only accounts for the node location and `queryStatement`, but it omits the retained sizes of `targetFileName` and `options`. Other AST nodes include string fields in their memory estimates; please add these fields here too to keep memory accounting accurate.
```suggestion
size += AstMemoryEstimationHelper.getEstimatedSizeOfAccountableObject(queryStatement);
size += AstMemoryEstimationHelper.getEstimatedSizeOfString(targetFileName);
size += AstMemoryEstimationHelper.getEstimatedSizeOfAccountableObject(options);
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/util/SqlFormatter.java:
##########
@@ -643,6 +644,21 @@ protected Void visitExplain(Explain node, Integer indent) {
return null;
}
+ @Override
+ protected Void visitCopyTo(CopyTo node, Integer context) {
+ builder.append("COPY\n");
+ builder.append("(\n");
+ process(node.getQueryStatement(), context);
+ builder.append("\n) ");
+ builder.append("TO ");
+ builder.append('\'');
+ builder.append(node.getTargetFileName());
+ builder.append('\'');
+ builder.append("\n");
+ builder.append(node.getOptions().toString());
Review Comment:
`SqlFormatter` currently appends `node.getOptions().toString()`, but `CopyToTsFileOptions.toString()` does not match the grammar keywords (e.g., it prints `TIME COLUMN`, `TAG COLUMNS`, `MEMORY` instead of `TIME`, `TAGS`, `MEMORY_THRESHOLD`). This makes formatted SQL invalid/unparseable and inconsistent with the actual user-facing syntax. Please format options explicitly here (or provide a dedicated formatter) using the grammar’s option keywords.
```suggestion
if (node.getOptions() != null) {
String optionsSql = node.getOptions().toString();
// Normalize option keywords to match the SQL grammar
optionsSql = optionsSql
.replace("TIME COLUMN", "TIME")
.replace("TAG COLUMNS", "TAGS")
.replace("MEMORY", "MEMORY_THRESHOLD");
builder.append(optionsSql);
}
```
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/copyto/tsfile/CopyToTsFileOptions.java:
##########
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.process.copyto.tsfile;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeader;
+import org.apache.iotdb.commons.schema.column.ColumnHeaderConstant;
+import org.apache.iotdb.commons.schema.table.TsTable;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnSchema;
+import org.apache.iotdb.db.exception.sql.SemanticException;
+import org.apache.iotdb.db.queryengine.execution.operator.process.copyto.CopyToOptions;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.Analysis;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.RelationPlan;
+import org.apache.iotdb.db.queryengine.plan.relational.sql.ast.Identifier;
+import org.apache.iotdb.db.schemaengine.table.DataNodeTableCache;
+import org.apache.iotdb.db.utils.constant.SqlConstant;
+
+import org.apache.tsfile.enums.TSDataType;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class CopyToTsFileOptions implements CopyToOptions {
+
+ private static final String DEFAULT_TABLE_NAME = "default";
+
+ private String targetTableName;
+ private String targetTimeColumn;
+ private Set<String> targetTagColumns;
+ private final long targetMemoryThreshold;
+
+ private boolean generateNewTimeColumn = false;
+
+ public CopyToTsFileOptions(
+ String targetTableName,
+ String targetTimeColumn,
+ Set<String> targetTagColumns,
+ long targetMemoryThreshold) {
+ this.targetTableName = targetTableName;
+ this.targetTimeColumn = targetTimeColumn;
+ this.targetTagColumns = targetTagColumns;
+ this.targetMemoryThreshold = targetMemoryThreshold;
+ }
+
+ public Format getFormat() {
+ return Format.TSFILE;
+ }
+
+ public boolean isGenerateNewTimeColumn() {
+ return generateNewTimeColumn;
+ }
+
+ @Override
+ public void infer(
+ Analysis analysis, RelationPlan queryRelationPlan, List<ColumnHeader> columnHeaders) {
+ List<Identifier> tables = queryRelationPlan.getScope().getTables();
+ TsTable onlyOneQueriedTable = null;
+ if (tables != null && tables.size() == 1) {
+ onlyOneQueriedTable =
+ DataNodeTableCache.getInstance()
+ .getTable(analysis.getDatabaseName(), tables.get(0).toString(), false);
+ }
+ if (targetTableName == null) {
+ targetTableName =
+ onlyOneQueriedTable == null ? DEFAULT_TABLE_NAME : onlyOneQueriedTable.getTableName();
+ }
+ if (onlyOneQueriedTable != null) {
+ if (targetTimeColumn == null || targetTagColumns == null) {
+ inferTimeAndTags(onlyOneQueriedTable, columnHeaders);
+ }
+ }
+ if (targetTimeColumn == null) {
+ generateNewTimeColumn = true;
+ targetTimeColumn = SqlConstant.TABLE_TIME_COLUMN_NAME;
+ }
+ if (targetTagColumns == null) {
+ targetTagColumns = Collections.emptySet();
+ }
+ }
+
+ private void inferTimeAndTags(TsTable tsTable, List<ColumnHeader> columnHeaders) {
+ Map<String, ColumnHeader> columnName2ColumnHeaderMapInDataset = new HashMap<>();
+ for (ColumnHeader columnHeader : columnHeaders) {
+ columnName2ColumnHeaderMapInDataset.put(columnHeader.getColumnName(), columnHeader);
+ }
+ if (targetTagColumns == null) {
+ boolean canMatchAllTags = true;
+ List<TsTableColumnSchema> tagColumnsInTsTable = tsTable.getTagColumnSchemaList();
+ for (TsTableColumnSchema tsTableColumnSchema : tagColumnsInTsTable) {
+ String columnName = tsTableColumnSchema.getColumnName();
+ ColumnHeader columnHeaderInDataset = columnName2ColumnHeaderMapInDataset.get(columnName);
+ if (columnHeaderInDataset == null
+ || columnHeaderInDataset.getColumnType() != tsTableColumnSchema.getDataType()) {
+ canMatchAllTags = false;
+ break;
+ }
+ }
+ if (canMatchAllTags) {
+ this.targetTagColumns = new LinkedHashSet<>(tagColumnsInTsTable.size());
+ for (TsTableColumnSchema tagColumn : tagColumnsInTsTable) {
+ targetTagColumns.add(tagColumn.getColumnName());
+ }
+ }
+ }
+
+ if (targetTimeColumn == null) {
+ String timeColumnInTsTable = tsTable.getTimeColumnName();
+ if (timeColumnInTsTable != null) {
+ ColumnHeader timeColumnHeader =
+ columnName2ColumnHeaderMapInDataset.get(timeColumnInTsTable);
+ if (timeColumnHeader != null) {
+ this.targetTimeColumn = timeColumnHeader.getColumnName();
+ }
+ }
+ if (targetTimeColumn == null
+ && columnName2ColumnHeaderMapInDataset.containsKey(SqlConstant.TABLE_TIME_COLUMN_NAME)) {
+ this.targetTimeColumn = SqlConstant.TABLE_TIME_COLUMN_NAME;
+ }
+ }
+ }
+
+ @Override
+ public void check(List<ColumnHeader> columnHeaders) {
+ if (generateNewTimeColumn && targetTagColumns.isEmpty()) {
+ return;
+ }
+ Set<String> columns = new HashSet<>(targetTagColumns.size());
+ int foundTagColumns = 0;
+ for (ColumnHeader columnHeader : columnHeaders) {
+ if (!generateNewTimeColumn
+ && columnHeader.getColumnName().equals(targetTimeColumn)
+ && columnHeader.getColumnType() != TSDataType.TIMESTAMP) {
+ throw new SemanticException("Data type of target time column is not TIMESTAMP");
+ }
+ if (targetTagColumns.contains(columnHeader.getColumnName())) {
+ if (columnHeader.getColumnType() != TSDataType.STRING) {
+ throw new SemanticException(
+ "Data type of tag column " + columnHeader.getColumnName() + " is not STRING");
+ }
+ foundTagColumns++;
+ }
+ columns.add(columnHeader.getColumnName());
+ }
+ if (columns.size() != columnHeaders.size()) {
+ throw new SemanticException("Duplicate column names in query dataset.");
+ }
+ if (foundTagColumns != targetTagColumns.size()) {
+ throw new SemanticException("Some specified tag columns are not exist in query dataset.");
+ }
+ if (foundTagColumns + (generateNewTimeColumn ? 0 : 1) == columns.size()) {
+ throw new SemanticException("Number of field columns should be larger than 0.");
+ }
+ }
+
+ public List<ColumnHeader> getRespColumnHeaders() {
+ return ColumnHeaderConstant.COPY_TO_TSFILE_COLUMN_HEADERS;
+ }
+
+ @Override
+ public long estimatedMaxRamBytesInWrite() {
+ return targetMemoryThreshold;
+ }
+
+ public String getTargetTableName() {
+ return targetTableName;
+ }
+
+ public String getTargetTimeColumn() {
+ return targetTimeColumn;
+ }
+
+ public Set<String> getTargetTagColumns() {
+ return targetTagColumns;
+ }
+
+ public long getTargetMemoryThreshold() {
+ return targetMemoryThreshold;
+ }
+
+ @Override
+ public String toString() {
+ return "("
+ +
+ // FORMAT
+ "\nFORMAT "
+ + getFormat().name()
+ +
+
+ // TABLE
+ "\nTABLE "
+ + targetTableName
+ +
+
+ // TIME COLUMN
+ "\nTIME COLUMN "
+ + targetTimeColumn
+ +
+
+ // TAG COLUMNS
+ "\nTAG COLUMNS ("
+ + String.join(", ", targetTagColumns)
+ + ")"
+ +
+
+ // MEMORY
+ "\nMEMORY "
Review Comment:
`toString()` renders option keywords that don’t match the COPY TO grammar (`TIME COLUMN`, `TAG COLUMNS`, `MEMORY` vs `TIME`, `TAGS`, `MEMORY_THRESHOLD`). Since SQL formatting/logging uses this output, it can be misleading and (when reused) unparseable. Align the rendered keywords with the grammar, or avoid using `toString()` for producing SQL text.
```suggestion
// TIME
"\nTIME "
+ targetTimeColumn
+
// TAGS
"\nTAGS ("
+ String.join(", ", targetTagColumns)
+ ")"
+
// MEMORY_THRESHOLD
"\nMEMORY_THRESHOLD "
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support COPY TO TsFile for exporting query results in table model [iotdb]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #17372:
URL: https://github.com/apache/iotdb/pull/17372#issuecomment-4141820835
## [](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372) **Quality Gate passed**
Issues
 [22 New issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_iotdb&pullRequest=17372&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb&pullRequest=17372&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_coverage&view=list)
 [3.5% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_iotdb&pullRequest=17372&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_iotdb&pullRequest=17372)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org