You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sohardforaname (via GitHub)" <gi...@apache.org> on 2023/04/20 11:09:52 UTC
[GitHub] [doris] sohardforaname opened a new pull request, #18869: [Feature](Nereids)support insert into select.
sohardforaname opened a new pull request, #18869:
URL: https://github.com/apache/doris/pull/18869
# Proposed changes
Issue Number: close #xxx
support insert the ret-value of a query into a table with `partition`, `with label`, `cols` tags:
Like：
```sql
insert into t partition (p1, p2) with label label_1 (c1, c2, c3) [hint1, hint2] with cte as (select * from src) select k1, k2, k3 from cte
```
We make class named InsertIntoTableCommand to describe the command. Add the insert into schema to statementCtx so than we can check if the query's outputs are compatible the target table schema.
When planned the sql, we create datasink in the run() in the command and init it. Then we run a transaction to insert the data.
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556697003
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1535868911
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556499053
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1519491728
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1537091691
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1537128269
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1559349842
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.04 seconds
stream load tsv: 417 seconds loaded 74807831229 Bytes, about 171 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 78.0 seconds inserted 10000000 Rows, about 128K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523132539_clickbench_pr_149083.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1560609832
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1561327810
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.47 seconds
stream load tsv: 418 seconds loaded 74807831229 Bytes, about 170 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 77.7 seconds inserted 10000000 Rows, about 128K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524150207_clickbench_pr_149913.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1189455545
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -209,6 +207,8 @@ public Plan plan(LogicalPlan plan, PhysicalProperties requireProperties, Explain
deriveStats();
+ adjustRequiredProperties(cascadesContext.getRewritePlan());
Review Comment:
because we may handle star which will be expanded in analyzer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1523057499
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1554099716
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1559141222
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1559287096
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.51 seconds
stream load tsv: 414 seconds loaded 74807831229 Bytes, about 172 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 78.5 seconds inserted 10000000 Rows, about 127K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523130232_clickbench_pr_149053.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1560783376
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.8 seconds
stream load tsv: 416 seconds loaded 74807831229 Bytes, about 171 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 79.0 seconds inserted 10000000 Rows, about 126K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524093627_clickbench_pr_149652.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1188511006
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -298,6 +298,23 @@ private void optimize() {
new CascadesOptimizer(cascadesContext).execute();
}
+ private void adjustRequiredProperties(Plan plan) {
+ PhysicalProperties properties = null;
+ if (statementContext.getParsedStatement() != null) {
+ Plan parsedStmt = ((LogicalPlanAdapter) statementContext.getParsedStatement()).getLogicalPlan();
+ if (parsedStmt instanceof InsertIntoTableCommand) {
+ properties = ((InsertIntoTableCommand) parsedStmt)
+ .calculatePhysicalProperties(plan.getOutput());
+ } else if (parsedStmt instanceof ExplainCommand) {
+ properties = PhysicalProperties.ANY;
+ }
Review Comment:
if it is a explain command, u should use the REAL PLAN in explain to generate required properties
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/Command.java:
##########
@@ -103,6 +103,11 @@ public boolean canBind() {
throw new RuntimeException("Command do not implement canResolve");
}
+ @Override
+ public boolean isCommand() {
+ return true;
+ }
+
Review Comment:
remove this interface
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
Review Comment:
```suggestion
// in original planner with txn model, we can execute sql like: insert into t select 1, 2, 3
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
+
+ txn.executeInsertIntoSelectCommand(executor);
+ }
+
+ private void checkDatabaseAndTable(ConnectContext ctx) {
+ List<String> qualifier = RelationUtil.getQualifierName(ctx, tableName);
+ String catalogName = qualifier.get(0);
+ String dbName = qualifier.get(1);
+ String tableName = qualifier.get(2);
+ CatalogIf catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+ if (catalog == null) {
+ throw new RuntimeException(String.format("Catalog %s does not exist.", catalogName));
+ }
+ try {
+ database = ((Database) catalog.getDb(dbName).orElseThrow(() ->
+ new RuntimeException("Database [" + dbName + "] does not exist.")));
+ table = database.getTable(tableName).orElseThrow(() ->
+ new RuntimeException("Table [" + tableName + "] does not exist in database [" + dbName + "]."));
Review Comment:
```suggestion
new AnalysisException("Table [" + tableName + "] does not exist in database [" + dbName + "]."));
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
Review Comment:
should handle all explain in ExplainCommand
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -209,6 +207,8 @@ public Plan plan(LogicalPlan plan, PhysicalProperties requireProperties, Explain
deriveStats();
+ adjustRequiredProperties(cascadesContext.getRewritePlan());
Review Comment:
why not do it at the beginning of `plan`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
Review Comment:
why no data insert? find the reason and add more info in this comment
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
Review Comment:
```suggestion
Preconditions.checkArgument(tableName != null, "tableName cannot be null in InsertIntoTableCommand");
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
Review Comment:
add a blank line before and after this line
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
Review Comment:
ExplainCommand(InsertIntoTableCommand(Query())) is more make sense
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/txn/InsertIntoContext.java:
##########
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.txn;
+
+import org.apache.doris.catalog.Column;
+
+import java.util.List;
+
+/**
+ * context for insert into command
+ */
+public class InsertIntoContext {
Review Comment:
maybe the better way to do that is add a new type node LogicalOlapTableSink.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
+
+ txn.executeInsertIntoSelectCommand(executor);
+ }
+
+ private void checkDatabaseAndTable(ConnectContext ctx) {
+ List<String> qualifier = RelationUtil.getQualifierName(ctx, tableName);
+ String catalogName = qualifier.get(0);
+ String dbName = qualifier.get(1);
+ String tableName = qualifier.get(2);
+ CatalogIf catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+ if (catalog == null) {
+ throw new RuntimeException(String.format("Catalog %s does not exist.", catalogName));
Review Comment:
```suggestion
throw new AnalysisException(String.format("Catalog %s does not exist.", catalogName));
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
Review Comment:
why only this attribute init as `null`?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
Review Comment:
```suggestion
public InsertIntoTableCommand(List<String> nameParts, String labelName, List<String> colNames,
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
Review Comment:
not use?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java:
##########
@@ -89,5 +89,6 @@ public enum PlanType {
COMMAND,
EXPLAIN_COMMAND,
- CREATE_POLICY_COMMAND
+ CREATE_POLICY_COMMAND,
+ INSERT_INTO_SELECT_COMMAND
Review Comment:
```suggestion
INSERT_INTO_TABLE_COMMAND
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
+
+ txn.executeInsertIntoSelectCommand(executor);
+ }
+
+ private void checkDatabaseAndTable(ConnectContext ctx) {
+ List<String> qualifier = RelationUtil.getQualifierName(ctx, tableName);
Review Comment:
```suggestion
List<String> qualifiedTableName = RelationUtil.getQualifierName(ctx, tableName);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
+
+ txn.executeInsertIntoSelectCommand(executor);
+ }
+
+ private void checkDatabaseAndTable(ConnectContext ctx) {
+ List<String> qualifier = RelationUtil.getQualifierName(ctx, tableName);
+ String catalogName = qualifier.get(0);
+ String dbName = qualifier.get(1);
+ String tableName = qualifier.get(2);
+ CatalogIf catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+ if (catalog == null) {
+ throw new RuntimeException(String.format("Catalog %s does not exist.", catalogName));
+ }
+ try {
+ database = ((Database) catalog.getDb(dbName).orElseThrow(() ->
+ new RuntimeException("Database [" + dbName + "] does not exist.")));
Review Comment:
```suggestion
new AnalysisException("Database [" + dbName + "] does not exist.")));
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
+
+ txn.executeInsertIntoSelectCommand(executor);
+ }
+
+ private void checkDatabaseAndTable(ConnectContext ctx) {
Review Comment:
bindTargetRelation
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
Review Comment:
put the code block back to `run` method and remove these functions
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/txn/Transaction.java:
##########
@@ -0,0 +1,243 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.txn;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableIf.TableType;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.load.EtlJobType;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.Coordinator;
+import org.apache.doris.qe.QeProcessorImpl;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.service.FrontendOptions;
+import org.apache.doris.task.LoadEtlTask;
+import org.apache.doris.thrift.TQueryType;
+import org.apache.doris.transaction.TabletCommitInfo;
+import org.apache.doris.transaction.TransactionState.LoadJobSourceType;
+import org.apache.doris.transaction.TransactionState.TxnCoordinator;
+import org.apache.doris.transaction.TransactionState.TxnSourceType;
+import org.apache.doris.transaction.TransactionStatus;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * transaction wrapper for Nereids
+ */
+public class Transaction {
Review Comment:
why need Nereids' transaction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1203909978
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundOlapTableSink.java:
##########
@@ -0,0 +1,130 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.UnboundLogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnary;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Represent an olap table sink plan node that has not been bound.
+ */
+public class UnboundOlapTableSink<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements Unbound {
+ private final List<String> nameParts;
+ private final List<String> colNames;
+ private final List<String> hints;
+ private final List<String> partitions;
+
+ public UnboundOlapTableSink(List<String> nameParts, List<String> colNames, List<String> hints,
+ List<String> partitions, CHILD_TYPE child) {
+ this(nameParts, colNames, hints, partitions, Optional.empty(), Optional.empty(), child);
+ }
+
+ public UnboundOlapTableSink(List<String> nameParts, List<String> colNames, List<String> hints,
+ List<String> partitions, Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
+ super(PlanType.LOGICAL_UNBOUND_OLAP_TABLE_SINK, groupExpression, logicalProperties, child);
+ this.nameParts = nameParts;
+ this.colNames = colNames;
+ this.hints = hints;
+ this.partitions = partitions;
Review Comment:
use ImmutableList.copyOf(Objects.requireNotNull(...))
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -95,6 +93,7 @@ public class NereidsPlanner extends Planner {
private Plan analyzedPlan;
private Plan rewrittenPlan;
private Plan optimizedPlan;
+ private PhysicalPlan physicalPlan;
Review Comment:
move back as local var
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -166,7 +165,6 @@ public PhysicalPlan plan(LogicalPlan plan, PhysicalProperties outputProperties)
* Do analyze and optimize for query plan.
*
* @param plan wait for plan
- * @param requireProperties request physical properties constraints
Review Comment:
add back this java doc
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -252,7 +250,7 @@ public Plan plan(LogicalPlan plan, PhysicalProperties requireProperties, Explain
LOG.info(memo);
}
- int nth = cascadesContext.getConnectContext().getSessionVariable().getNthOptimizedPlan();
+ int nth = ConnectContext.get().getSessionVariable().getNthOptimizedPlan();
Review Comment:
change back to cascadesContext.getConnectContext()
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.statistics.Statistics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * physical olap table sink for insert command
+ */
+public class PhysicalOlapTableSink<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> {
+ private final Database database;
+ private final OlapTable targetTable;
+ private final List<Column> cols;
+ private final List<Long> partitionIds;
+ private final boolean singleReplicaLoad;
+
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, LogicalProperties logicalProperties, CHILD_TYPE child) {
+ this(database, targetTable, partitionIds, cols, singleReplicaLoad, Optional.empty(), logicalProperties, child);
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, physicalProperties,
+ statistics, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ public Database getDatabase() {
+ return database;
+ }
+
+ public OlapTable getTargetTable() {
+ return targetTable;
+ }
+
+ public List<Column> getCols() {
+ return cols;
+ }
+
+ public List<Long> getPartitionIds() {
+ return partitionIds;
+ }
+
+ public boolean isSingleReplicaLoad() {
+ return singleReplicaLoad;
+ }
+
+ @Override
+ public Plan withChildren(List<Plan> children) {
+ Preconditions.checkArgument(children.size() == 1, "PhysicalOlapTableSink only accepts one child");
+ return new PhysicalOlapTableSink<>(database, targetTable, partitionIds, cols, singleReplicaLoad,
+ getLogicalProperties(), children.get(0));
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
+ return false;
+ }
+ PhysicalOlapTableSink<?> that = (PhysicalOlapTableSink<?>) o;
+ return Objects.equals(targetTable, that.targetTable) && Objects.equals(partitionIds,
+ that.partitionIds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(targetTable, partitionIds);
+ }
+
+ @Override
+ public List<Slot> computeOutput() {
+ return child().getOutput();
+ }
+
+ @Override
+ public List<Slot> getOutput() {
+ return computeOutput();
+ }
+
+ @Override
+ public Set<Slot> getOutputSet() {
Review Comment:
override which base class's function?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.statistics.Statistics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * physical olap table sink for insert command
+ */
+public class PhysicalOlapTableSink<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> {
+ private final Database database;
+ private final OlapTable targetTable;
+ private final List<Column> cols;
+ private final List<Long> partitionIds;
+ private final boolean singleReplicaLoad;
+
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, LogicalProperties logicalProperties, CHILD_TYPE child) {
+ this(database, targetTable, partitionIds, cols, singleReplicaLoad, Optional.empty(), logicalProperties, child);
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, physicalProperties,
+ statistics, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ public Database getDatabase() {
+ return database;
+ }
+
+ public OlapTable getTargetTable() {
+ return targetTable;
+ }
+
+ public List<Column> getCols() {
+ return cols;
+ }
+
+ public List<Long> getPartitionIds() {
+ return partitionIds;
+ }
+
+ public boolean isSingleReplicaLoad() {
+ return singleReplicaLoad;
+ }
+
+ @Override
+ public Plan withChildren(List<Plan> children) {
+ Preconditions.checkArgument(children.size() == 1, "PhysicalOlapTableSink only accepts one child");
+ return new PhysicalOlapTableSink<>(database, targetTable, partitionIds, cols, singleReplicaLoad,
+ getLogicalProperties(), children.get(0));
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
+ return false;
+ }
+ PhysicalOlapTableSink<?> that = (PhysicalOlapTableSink<?>) o;
+ return Objects.equals(targetTable, that.targetTable) && Objects.equals(partitionIds,
+ that.partitionIds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(targetTable, partitionIds);
Review Comment:
why only table and partition in equals and hashCode?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.statistics.Statistics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * physical olap table sink for insert command
+ */
+public class PhysicalOlapTableSink<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> {
+ private final Database database;
+ private final OlapTable targetTable;
+ private final List<Column> cols;
+ private final List<Long> partitionIds;
+ private final boolean singleReplicaLoad;
+
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, LogicalProperties logicalProperties, CHILD_TYPE child) {
+ this(database, targetTable, partitionIds, cols, singleReplicaLoad, Optional.empty(), logicalProperties, child);
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, physicalProperties,
+ statistics, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
Review Comment:
ditto
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -284,6 +291,82 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan) {
return rootFragment;
}
+ @Override
+ public PlanFragment visitPhysicalOlapTableSink(PhysicalOlapTableSink<? extends Plan> olapTableSink,
+ PlanTranslatorContext context) {
+ PlanFragment rootFragment = olapTableSink.child().accept(this, context);
+
+ TupleDescriptor olapTuple = context.generateTupleDesc();
+ for (Column column : olapTableSink.getTargetTable().getFullSchema()) {
+ SlotDescriptor slotDesc = context.addSlotDesc(olapTuple);
+ slotDesc.setIsMaterialized(true);
+ slotDesc.setType(column.getType());
+ slotDesc.setColumn(column);
+ slotDesc.setIsNullable(column.isAllowNull());
+ }
+
+ OlapTableSink sink = new OlapTableSink(
+ olapTableSink.getTargetTable(),
+ olapTuple,
+ olapTableSink.getPartitionIds(),
+ olapTableSink.isSingleReplicaLoad()
+ );
+
+ Map<Column, Slot> columnToSlots = Maps.newHashMap();
+ Preconditions.checkArgument(olapTableSink.getOutput().size() == olapTableSink.getCols().size(),
+ "this is a bug in insert into command");
+ for (int i = 0; i < olapTableSink.getCols().size(); ++i) {
+ columnToSlots.put(olapTableSink.getCols().get(i), olapTableSink.getOutput().get(i));
+ }
+ List<Expr> outputExprs = Lists.newArrayList();
+ try {
+ for (Column column : olapTableSink.getTargetTable().getFullSchema()) {
+ if (columnToSlots.containsKey(column)) {
+ ExprId exprId = columnToSlots.get(column).getExprId();
+ outputExprs.add(context.findSlotRef(exprId).checkTypeCompatibility(column.getType()));
+ } else if (column.getDefaultValue() == null) {
+ outputExprs.add(NullLiteral.create(column.getType()));
+ } else {
+ if (column.getDefaultValueExprDef() != null) {
+ outputExprs.add(column.getDefaultValueExpr());
+ } else {
+ StringLiteral defaultValueExpr = new StringLiteral(column.getDefaultValue());
+ outputExprs.add(defaultValueExpr.checkTypeCompatibility(column.getType()));
+ }
+ }
+ }
+ } catch (Exception e) {
+ throw new AnalysisException(e.getMessage(), e.getCause());
+ }
+
+ HashDistributionInfo distributionInfo = ((HashDistributionInfo) olapTableSink.getTargetTable()
+ .getDefaultDistributionInfo());
+ List<Expr> partitionExprs = distributionInfo.getDistributionColumns().stream()
+ .map(column -> context.findSlotRef(columnToSlots.get(column).getExprId()))
+ .collect(Collectors.toList());
+
+ if (rootFragment.getPlanRoot() instanceof ExchangeNode) {
+ ExchangeNode exchangeNode = ((ExchangeNode) rootFragment.getPlanRoot());
+ PlanFragment currentFragment = new PlanFragment(
+ context.nextFragmentId(),
+ exchangeNode,
+ rootFragment.getDataPartition());
+
+ rootFragment.setPlanRoot(exchangeNode.getChild(0));
+ rootFragment.getPlanRoot().setNumInstances(1);
+ rootFragment.setDestination(exchangeNode);
Review Comment:
so, just set data partition and remove the useless exchange?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.statistics.Statistics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * physical olap table sink for insert command
+ */
+public class PhysicalOlapTableSink<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> {
+ private final Database database;
+ private final OlapTable targetTable;
+ private final List<Column> cols;
+ private final List<Long> partitionIds;
+ private final boolean singleReplicaLoad;
+
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, LogicalProperties logicalProperties, CHILD_TYPE child) {
+ this(database, targetTable, partitionIds, cols, singleReplicaLoad, Optional.empty(), logicalProperties, child);
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ PhysicalProperties physicalProperties, Statistics statistics, CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, physicalProperties,
+ statistics, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
+ this.singleReplicaLoad = singleReplicaLoad;
+ }
+
+ public Database getDatabase() {
+ return database;
+ }
+
+ public OlapTable getTargetTable() {
+ return targetTable;
+ }
+
+ public List<Column> getCols() {
+ return cols;
+ }
+
+ public List<Long> getPartitionIds() {
+ return partitionIds;
+ }
+
+ public boolean isSingleReplicaLoad() {
+ return singleReplicaLoad;
+ }
+
+ @Override
+ public Plan withChildren(List<Plan> children) {
+ Preconditions.checkArgument(children.size() == 1, "PhysicalOlapTableSink only accepts one child");
+ return new PhysicalOlapTableSink<>(database, targetTable, partitionIds, cols, singleReplicaLoad,
+ getLogicalProperties(), children.get(0));
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ if (!super.equals(o)) {
+ return false;
+ }
+ PhysicalOlapTableSink<?> that = (PhysicalOlapTableSink<?>) o;
+ return Objects.equals(targetTable, that.targetTable) && Objects.equals(partitionIds,
+ that.partitionIds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(targetTable, partitionIds);
+ }
+
+ @Override
+ public List<Slot> computeOutput() {
+ return child().getOutput();
+ }
+
+ @Override
+ public List<Slot> getOutput() {
+ return computeOutput();
+ }
+
+ @Override
+ public Set<Slot> getOutputSet() {
+ return ImmutableSet.copyOf(getOutput());
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitPhysicalOlapTableSink(this, context);
+ }
+
+ @Override
+ public List<? extends Expression> getExpressions() {
+ return ImmutableList.of();
+ }
+
+ @Override
+ public Plan withGroupExpression(Optional<GroupExpression> groupExpression) {
+ return new PhysicalOlapTableSink<>(database, targetTable, partitionIds, cols, singleReplicaLoad,
+ groupExpression, getLogicalProperties(), child());
+ }
+
+ @Override
+ public Plan withLogicalProperties(Optional<LogicalProperties> logicalProperties) {
+ return new PhysicalOlapTableSink<>(database, targetTable, partitionIds, cols, singleReplicaLoad,
+ groupExpression, logicalProperties.get(), child());
+ }
+
+ @Override
+ public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalProperties, Statistics statistics) {
+ return new PhysicalOlapTableSink<>(database, targetTable, partitionIds, cols, singleReplicaLoad,
+ groupExpression, getLogicalProperties(), physicalProperties, statistics, child());
+ }
+
+ /**
+ * get output physical properties
+ */
+ public PhysicalProperties getRequirePhysicalProperties() {
+ HashDistributionInfo distributionInfo = ((HashDistributionInfo) targetTable.getDefaultDistributionInfo());
+ List<Column> distributedColumns = distributionInfo.getDistributionColumns();
+ List<Integer> columnIndexes = Lists.newArrayList();
+ int idx = 0;
+ for (int i = 0; i < targetTable.getFullSchema().size(); ++i) {
+ if (targetTable.getFullSchema().get(i).equals(distributedColumns.get(idx))) {
+ columnIndexes.add(i);
+ idx++;
+ if (idx == distributedColumns.size()) {
+ break;
+ }
+ }
+ }
+ return PhysicalProperties.createHash(columnIndexes.stream()
+ .map(colIdx -> getOutput().get(colIdx).getExprId()).collect(Collectors.toList()), ShuffleType.NATURAL);
Review Comment:
why the require properties type is NATRUAL? NATRUAL is only use for output
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -284,6 +291,82 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan) {
return rootFragment;
}
+ @Override
+ public PlanFragment visitPhysicalOlapTableSink(PhysicalOlapTableSink<? extends Plan> olapTableSink,
+ PlanTranslatorContext context) {
+ PlanFragment rootFragment = olapTableSink.child().accept(this, context);
+
+ TupleDescriptor olapTuple = context.generateTupleDesc();
+ for (Column column : olapTableSink.getTargetTable().getFullSchema()) {
+ SlotDescriptor slotDesc = context.addSlotDesc(olapTuple);
+ slotDesc.setIsMaterialized(true);
+ slotDesc.setType(column.getType());
+ slotDesc.setColumn(column);
+ slotDesc.setIsNullable(column.isAllowNull());
+ }
+
+ OlapTableSink sink = new OlapTableSink(
+ olapTableSink.getTargetTable(),
+ olapTuple,
+ olapTableSink.getPartitionIds(),
+ olapTableSink.isSingleReplicaLoad()
+ );
+
+ Map<Column, Slot> columnToSlots = Maps.newHashMap();
+ Preconditions.checkArgument(olapTableSink.getOutput().size() == olapTableSink.getCols().size(),
+ "this is a bug in insert into command");
+ for (int i = 0; i < olapTableSink.getCols().size(); ++i) {
+ columnToSlots.put(olapTableSink.getCols().get(i), olapTableSink.getOutput().get(i));
+ }
+ List<Expr> outputExprs = Lists.newArrayList();
+ try {
+ for (Column column : olapTableSink.getTargetTable().getFullSchema()) {
+ if (columnToSlots.containsKey(column)) {
+ ExprId exprId = columnToSlots.get(column).getExprId();
+ outputExprs.add(context.findSlotRef(exprId).checkTypeCompatibility(column.getType()));
+ } else if (column.getDefaultValue() == null) {
+ outputExprs.add(NullLiteral.create(column.getType()));
+ } else {
+ if (column.getDefaultValueExprDef() != null) {
+ outputExprs.add(column.getDefaultValueExpr());
+ } else {
+ StringLiteral defaultValueExpr = new StringLiteral(column.getDefaultValue());
+ outputExprs.add(defaultValueExpr.checkTypeCompatibility(column.getType()));
+ }
+ }
+ }
+ } catch (Exception e) {
+ throw new AnalysisException(e.getMessage(), e.getCause());
+ }
+
+ HashDistributionInfo distributionInfo = ((HashDistributionInfo) olapTableSink.getTargetTable()
+ .getDefaultDistributionInfo());
+ List<Expr> partitionExprs = distributionInfo.getDistributionColumns().stream()
+ .map(column -> context.findSlotRef(columnToSlots.get(column).getExprId()))
+ .collect(Collectors.toList());
+
+ if (rootFragment.getPlanRoot() instanceof ExchangeNode) {
+ ExchangeNode exchangeNode = ((ExchangeNode) rootFragment.getPlanRoot());
+ PlanFragment currentFragment = new PlanFragment(
+ context.nextFragmentId(),
+ exchangeNode,
+ rootFragment.getDataPartition());
+
+ rootFragment.setPlanRoot(exchangeNode.getChild(0));
+ rootFragment.getPlanRoot().setNumInstances(1);
+ rootFragment.setDestination(exchangeNode);
+ context.addPlanFragment(currentFragment);
+ rootFragment = currentFragment;
+ }
+ rootFragment.getPlanRoot().setNumInstances(1);
Review Comment:
why need to set this?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalOlapTableSink.java:
##########
@@ -0,0 +1,201 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.HashDistributionInfo;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.statistics.Statistics;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * physical olap table sink for insert command
+ */
+public class PhysicalOlapTableSink<CHILD_TYPE extends Plan> extends PhysicalUnary<CHILD_TYPE> {
+ private final Database database;
+ private final OlapTable targetTable;
+ private final List<Column> cols;
+ private final List<Long> partitionIds;
+ private final boolean singleReplicaLoad;
+
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, LogicalProperties logicalProperties, CHILD_TYPE child) {
+ this(database, targetTable, partitionIds, cols, singleReplicaLoad, Optional.empty(), logicalProperties, child);
+ }
+
+ /**
+ * Constructor
+ */
+ public PhysicalOlapTableSink(Database database, OlapTable targetTable, List<Long> partitionIds, List<Column> cols,
+ boolean singleReplicaLoad, Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(PlanType.PHYSICAL_OLAP_TABLE_SINK, groupExpression, logicalProperties, child);
+ this.database = Preconditions.checkNotNull(database, "database != null in PhysicalOlapTableSink");
+ this.targetTable = Preconditions.checkNotNull(targetTable, "targetTable != null in PhysicalOlapTableSink");
+ this.cols = cols;
+ this.partitionIds = partitionIds;
Review Comment:
ImmutableList.copyOf(Objects.requireNotNull(...))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556884198
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556773761
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1561489921
do not merge this PR until the author update commit msg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1526919879
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1519522466
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.22 seconds
stream load tsv: 423 seconds loaded 74807831229 Bytes, about 168 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230424072025_clickbench_pr_133873.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1535868747
run fe ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1537134725
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1192028320
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/txn/Transaction.java:
##########
@@ -0,0 +1,243 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.txn;
+
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.catalog.TableIf.TableType;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.ErrorReport;
+import org.apache.doris.common.MetaNotFoundException;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.load.EtlJobType;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.Coordinator;
+import org.apache.doris.qe.QeProcessorImpl;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.service.FrontendOptions;
+import org.apache.doris.task.LoadEtlTask;
+import org.apache.doris.thrift.TQueryType;
+import org.apache.doris.transaction.TabletCommitInfo;
+import org.apache.doris.transaction.TransactionState.LoadJobSourceType;
+import org.apache.doris.transaction.TransactionState.TxnCoordinator;
+import org.apache.doris.transaction.TransactionState.TxnSourceType;
+import org.apache.doris.transaction.TransactionStatus;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * transaction wrapper for Nereids
+ */
+public class Transaction {
Review Comment:
because the handleInsertStmt() in original planner is too hard to be rewritten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1561071134
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556634024
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1560592739
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.89 seconds
stream load tsv: 420 seconds loaded 74807831229 Bytes, about 169 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 79.1 seconds inserted 10000000 Rows, about 126K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230524072841_clickbench_pr_149576.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556458076
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1561464692
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1204031381
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##########
@@ -95,6 +93,7 @@ public class NereidsPlanner extends Planner {
private Plan analyzedPlan;
private Plan rewrittenPlan;
private Plan optimizedPlan;
+ private PhysicalPlan physicalPlan;
Review Comment:
no, will be used in insert into table command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558503663
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.68 seconds
stream load tsv: 418 seconds loaded 74807831229 Bytes, about 170 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 76.6 seconds inserted 10000000 Rows, about 130K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523043006_clickbench_pr_148716.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556641681
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558871723
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.58 seconds
stream load tsv: 415 seconds loaded 74807831229 Bytes, about 171 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 78.4 seconds inserted 10000000 Rows, about 127K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523091037_clickbench_pr_148848.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1559277967
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1521215112
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1176184881
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SetRequiredProperties.java:
##########
@@ -0,0 +1,66 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
+import org.apache.doris.nereids.trees.plans.commands.InsertIntoTableCommand;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * adjust the required properties(which is null before here) to a meaningful value.
+ */
+public class SetRequiredProperties implements CustomRewriter {
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ CascadesContext cascadesCtx = jobContext.getCascadesContext();
+ StatementContext stmtCtx = jobContext.getCascadesContext().getStatementContext();
+ if (stmtCtx.getParsedStatement() != null) {
+ Plan parsedStmt = ((LogicalPlanAdapter) stmtCtx.getParsedStatement()).getLogicalPlan();
+ if (parsedStmt instanceof ExplainCommand) {
+ cascadesCtx.setJobContext(PhysicalProperties.ANY);
+ } else if (parsedStmt instanceof InsertIntoTableCommand) {
+ Preconditions.checkArgument(stmtCtx.getInsertIntoContext() != null, "insert into"
+ + " table context is null when running insert into table command");
+ int keysNum = stmtCtx.getInsertIntoContext().getKeyNums();
+ List<ExprId> outputs = plan.getOutput().subList(0, keysNum).stream()
+ .map(NamedExpression::getExprId).collect(Collectors.toList());
+ cascadesCtx.setJobContext(PhysicalProperties.createHash(
+ new DistributionSpecHash(outputs, ShuffleType.NATURAL)));
+ }
+ }
+ if (cascadesCtx.getCurrentJobContext().getRequiredProperties() == null) {
+ cascadesCtx.setJobContext(PhysicalProperties.GATHER);
+ }
Review Comment:
Maybe this is not the good position to set require properties, because the InsertIntoTableCommand set hash require properties for the select part, and any require for the insert part.
And you process these 3 cases:
1. if is ExplainCommand: you should expend ExplainCommand and according the inner statement to set required properties;
2. if is InsertIntoTableCommand, hash require is right, but I don't know whether ShuffleType.NATURAL is the true require, @morrySnow PTAL
3. you are right, else cases use gather require for query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1520106835
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1516381077
A known bug: insert into unique table without group by unique key will cause the replication different.
3 replications maybe receive the data with different when exchange.
e.g.
```
+-----------+ +--------------------------+
| machine 1 | exchange | machine 4 (replica 1) |
+-----------+ +--------------------------+
| 'a' | | 'a', 'b', 'c' |
+-----------+ +--------------------------+
\
+-----------+ +--------------------------+
| machine 2 | | machine 5 (replica 2) |
+-----------+ - +--------------------------+
| 'b' | | 'c', 'a', 'b' |
+-----------+ +--------------------------+
/
+-----------+ +--------------------------+
| machine 3 | | machine 5 (replica 2) |
+-----------+ +--------------------------+
| 'c' | | 'b', 'c', 'a' |
+-----------+ +--------------------------+
```
so when insert into the table with unique key, we should let the final instance satisfy the distribution same with the key, and you should add a hash(unique key) require properties to the top of the select
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1176184881
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SetRequiredProperties.java:
##########
@@ -0,0 +1,66 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
+import org.apache.doris.nereids.trees.plans.commands.InsertIntoTableCommand;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * adjust the required properties(which is null before here) to a meaningful value.
+ */
+public class SetRequiredProperties implements CustomRewriter {
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ CascadesContext cascadesCtx = jobContext.getCascadesContext();
+ StatementContext stmtCtx = jobContext.getCascadesContext().getStatementContext();
+ if (stmtCtx.getParsedStatement() != null) {
+ Plan parsedStmt = ((LogicalPlanAdapter) stmtCtx.getParsedStatement()).getLogicalPlan();
+ if (parsedStmt instanceof ExplainCommand) {
+ cascadesCtx.setJobContext(PhysicalProperties.ANY);
+ } else if (parsedStmt instanceof InsertIntoTableCommand) {
+ Preconditions.checkArgument(stmtCtx.getInsertIntoContext() != null, "insert into"
+ + " table context is null when running insert into table command");
+ int keysNum = stmtCtx.getInsertIntoContext().getKeyNums();
+ List<ExprId> outputs = plan.getOutput().subList(0, keysNum).stream()
+ .map(NamedExpression::getExprId).collect(Collectors.toList());
+ cascadesCtx.setJobContext(PhysicalProperties.createHash(
+ new DistributionSpecHash(outputs, ShuffleType.NATURAL)));
+ }
+ }
+ if (cascadesCtx.getCurrentJobContext().getRequiredProperties() == null) {
+ cascadesCtx.setJobContext(PhysicalProperties.GATHER);
+ }
Review Comment:
1. if is ExplainCommand: you should expend ExplainCommand and according the inner statement to set required properties;
2. if is InsertIntoTableCommand, hash require is right, but I don't know whether ShuffleType.NATURAL is the true require, @morrySnow PTAL
3. you are right, else cases use gather require for query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556733470
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558485868
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558830104
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1559202999
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.73 seconds
stream load tsv: 419 seconds loaded 74807831229 Bytes, about 170 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 77.6 seconds inserted 10000000 Rows, about 128K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523122834_clickbench_pr_149042.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1559150396
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1560539293
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1552819037
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556456774
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558421349
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558450276
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.3 seconds
stream load tsv: 418 seconds loaded 74807831229 Bytes, about 170 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 78.4 seconds inserted 10000000 Rows, about 127K ops/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230523031506_clickbench_pr_148616.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558596069
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #18869: [Feature](Nereids)support insert into select.
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #18869:
URL: https://github.com/apache/doris/pull/18869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1176184881
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SetRequiredProperties.java:
##########
@@ -0,0 +1,66 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.commands.ExplainCommand;
+import org.apache.doris.nereids.trees.plans.commands.InsertIntoTableCommand;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * adjust the required properties(which is null before here) to a meaningful value.
+ */
+public class SetRequiredProperties implements CustomRewriter {
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ CascadesContext cascadesCtx = jobContext.getCascadesContext();
+ StatementContext stmtCtx = jobContext.getCascadesContext().getStatementContext();
+ if (stmtCtx.getParsedStatement() != null) {
+ Plan parsedStmt = ((LogicalPlanAdapter) stmtCtx.getParsedStatement()).getLogicalPlan();
+ if (parsedStmt instanceof ExplainCommand) {
+ cascadesCtx.setJobContext(PhysicalProperties.ANY);
+ } else if (parsedStmt instanceof InsertIntoTableCommand) {
+ Preconditions.checkArgument(stmtCtx.getInsertIntoContext() != null, "insert into"
+ + " table context is null when running insert into table command");
+ int keysNum = stmtCtx.getInsertIntoContext().getKeyNums();
+ List<ExprId> outputs = plan.getOutput().subList(0, keysNum).stream()
+ .map(NamedExpression::getExprId).collect(Collectors.toList());
+ cascadesCtx.setJobContext(PhysicalProperties.createHash(
+ new DistributionSpecHash(outputs, ShuffleType.NATURAL)));
+ }
+ }
+ if (cascadesCtx.getCurrentJobContext().getRequiredProperties() == null) {
+ cascadesCtx.setJobContext(PhysicalProperties.GATHER);
+ }
Review Comment:
Maybe this is not the good position to set require properties, because the InsertIntoTableCommand set hash require properties for the select part, and any require for the insert part.
1. if is ExplainCommand: you should expend ExplainCommand and according the inner statement to set required properties;
2. if is InsertIntoTableCommand, hash require is right, but I don't know whether ShuffleType.NATURAL is the true require, @morrySnow PTAL
3. you are right, else cases use gather require for query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1519494126
@924060929 fixed the bug you mentioned, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1536202787
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1539996932
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1520046078
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1521797586
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1192078375
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,271 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotDescriptor;
+import org.apache.doris.analysis.TupleDescriptor;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.Partition;
+import org.apache.doris.catalog.Table;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.properties.DistributionSpecHash;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * insert into select command implementation
+ *
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * InsertIntoTableCommand(ExplainCommand(Query()))
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+ private final List<String> tableName;
+ private final List<String> colNames;
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private Database database;
+ private Table table;
+ private NereidsPlanner planner;
+ private TupleDescriptor olapTuple;
+ private List<String> partitions;
+ private List<String> hints;
+ private List<Column> targetColumns;
+ private List<Long> partitionIds = null;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(List<String> tableName, String labelName, List<String> colNames,
+ List<String> partitions, List<String> hints, LogicalPlan logicalQuery) {
+ super(PlanType.INSERT_INTO_SELECT_COMMAND);
+ Preconditions.checkArgument(tableName != null, "tableName cannot be null in insert-into-select command");
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in insert-into-select command");
+ this.tableName = tableName;
+ this.labelName = labelName;
+ this.colNames = colNames;
+ this.partitions = partitions;
+ this.hints = hints;
+ this.logicalQuery = logicalQuery;
+ }
+
+ public NereidsPlanner getPlanner() {
+ return planner;
+ }
+
+ @Override
+ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
+ if (ctx.isTxnModel()) {
+ // in original planner and in txn model, we can execute sql like: insert into t select 1, 2, 3
+ // but no data will be inserted, now we adjust forbid it.
+ throw new AnalysisException("insert into table command is not supported in txn model");
+ }
+ checkDatabaseAndTable(ctx);
+ getColumns();
+ getPartition();
+
+ ctx.getStatementContext().getInsertIntoContext().setTargetSchema(targetColumns);
+
+ LogicalPlanAdapter logicalPlanAdapter = new LogicalPlanAdapter(extractPlan(logicalQuery),
+ ctx.getStatementContext());
+ planner = new NereidsPlanner(ctx.getStatementContext());
+ planner.plan(logicalPlanAdapter, ctx.getSessionVariable().toThrift());
+
+ getTupleDesc();
+ addUnassignedColumns();
+
+ if (ctx.getMysqlChannel() != null) {
+ ctx.getMysqlChannel().reset();
+ }
+ String label = this.labelName;
+ if (label == null) {
+ label = String.format("label_%x_%x", ctx.queryId().hi, ctx.queryId().lo);
+ }
+
+ Transaction txn;
+ PlanFragment root = planner.getFragments().get(0);
+ DataSink sink = createDataSink(ctx, root);
+ Preconditions.checkArgument(sink instanceof OlapTableSink, "olap table sink is expected when"
+ + " running insert into select");
+ txn = new Transaction(ctx, database, table, label, planner);
+
+ OlapTableSink olapTableSink = ((OlapTableSink) sink);
+ olapTableSink.init(ctx.queryId(), txn.getTxnId(), database.getId(), ctx.getExecTimeout(),
+ ctx.getSessionVariable().getSendBatchParallelism(), false);
+ olapTableSink.complete();
+ root.resetSink(olapTableSink);
+
+ if (isExplain()) {
+ executor.handleExplainStmt(((ExplainCommand) logicalQuery).getExplainString(planner));
+ return;
+ }
+
+ txn.executeInsertIntoSelectCommand(executor);
+ }
+
+ private void checkDatabaseAndTable(ConnectContext ctx) {
Review Comment:
What does it mean？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1523257354
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1541212477
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #18869: [Feature](Nereids)support insert into select.
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #18869:
URL: https://github.com/apache/doris/pull/18869#discussion_r1201697158
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.util.ProfileManager.ProfileType;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.QueryState.MysqlStateType;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.transaction.TransactionState;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * insert into select command implementation
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * ExplainCommand(Query())
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
Review Comment:
add a blank line before this line
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java:
##########
@@ -0,0 +1,140 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.util.ProfileManager.ProfileType;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.glue.LogicalPlanAdapter;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.txn.Transaction;
+import org.apache.doris.planner.OlapTableSink;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.QueryState.MysqlStateType;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.transaction.TransactionState;
+
+import com.google.common.base.Preconditions;
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * insert into select command implementation
+ * insert into select command support the grammer: explain? insert into table columns? partitions? hints? query
+ * InsertIntoTableCommand is a command to represent insert the answer of a query into a table.
+ * class structure's:
+ * InsertIntoTableCommand(Query())
+ * ExplainCommand(Query())
+ */
+public class InsertIntoTableCommand extends Command implements ForwardWithSync {
+ public static final Logger LOG = LogManager.getLogger(InsertIntoTableCommand.class);
+
+ private final LogicalPlan logicalQuery;
+ private final String labelName;
+ private NereidsPlanner planner;
+ private boolean isTxnBegin = false;
+
+ /**
+ * constructor
+ */
+ public InsertIntoTableCommand(LogicalPlan logicalQuery, String labelName) {
+ super(PlanType.INSERT_INTO_TABLE_COMMAND);
+ Preconditions.checkArgument(logicalQuery != null, "logicalQuery cannot be null in InsertIntoTableCommand");
Review Comment:
Preconditions.checkNotNull
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExplainCommand.java:
##########
Review Comment:
revert these changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1554445767
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1554265033
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1556925645
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "sohardforaname (via GitHub)" <gi...@apache.org>.
sohardforaname commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1558687837
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #18869: [Feature](Nereids)support insert into select.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #18869:
URL: https://github.com/apache/doris/pull/18869#issuecomment-1561464820
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org