You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wuwenchi (via GitHub)" <gi...@apache.org> on 2024/04/28 12:14:29 UTC
[PR] [feature](iceberg)support iceberg insert grammar [doris]
wuwenchi opened a new pull request, #34257:
URL: https://github.com/apache/doris/pull/34257
## Proposed changes
Issue #31442
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2081848053
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2114192351
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2116624764
<details>
<summary>TPC-DS: <b>Total hot run time: 187209 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86a406858bf5f0101df4a8e0be10483dbaa10b44, data reload: false
query1 917 397 367 367
query2 6240 2397 2168 2168
query3 6653 209 209 209
query4 23234 21752 21988 21752
query5 3835 445 422 422
query6 280 189 173 173
query7 4546 292 292 292
query8 236 184 207 184
query9 8442 2418 2403 2403
query10 408 269 242 242
query11 15438 14859 14823 14823
query12 122 85 86 85
query13 1605 393 364 364
query14 9433 8324 6774 6774
query15 241 172 163 163
query16 8148 254 248 248
query17 1731 561 548 548
query18 2097 278 283 278
query19 232 143 145 143
query20 86 81 83 81
query21 197 126 123 123
query22 5131 4884 4810 4810
query23 34377 33666 33640 33640
query24 10865 2889 2919 2889
query25 610 364 368 364
query26 1306 158 150 150
query27 3056 321 314 314
query28 7275 2004 2011 2004
query29 857 642 600 600
query30 237 171 169 169
query31 987 741 736 736
query32 91 48 52 48
query33 734 250 241 241
query34 1057 472 471 471
query35 803 664 676 664
query36 1072 965 928 928
query37 144 67 70 67
query38 2905 2763 2807 2763
query39 1609 1563 1552 1552
query40 203 122 123 122
query41 45 43 48 43
query42 98 100 98 98
query43 576 563 519 519
query44 1187 716 733 716
query45 261 254 252 252
query46 1077 722 722 722
query47 1970 1860 1877 1860
query48 366 300 299 299
query49 881 417 380 380
query50 745 380 371 371
query51 6814 6648 6697 6648
query52 99 94 90 90
query53 353 276 277 276
query54 860 418 430 418
query55 78 70 72 70
query56 235 211 223 211
query57 1220 1151 1147 1147
query58 216 210 196 196
query59 3367 3308 3162 3162
query60 255 229 237 229
query61 90 91 86 86
query62 669 473 467 467
query63 310 278 283 278
query64 8738 7352 7365 7352
query65 3146 3100 3121 3100
query66 1370 326 343 326
query67 15707 15124 15060 15060
query68 4609 523 528 523
query69 525 300 310 300
query70 1190 1137 1144 1137
query71 428 275 264 264
query72 8099 2590 2413 2413
query73 719 315 321 315
query74 6514 6075 6183 6075
query75 3649 2660 2610 2610
query76 3324 989 1004 989
query77 573 261 265 261
query78 10573 10176 10103 10103
query79 2476 508 504 504
query80 1091 435 430 430
query81 540 246 247 246
query82 1335 94 93 93
query83 273 167 166 166
query84 242 96 83 83
query85 1532 263 258 258
query86 500 308 325 308
query87 3302 3111 3101 3101
query88 3946 2285 2286 2285
query89 457 377 375 375
query90 1941 186 184 184
query91 126 97 98 97
query92 60 51 48 48
query93 2144 487 482 482
query94 1193 185 184 184
query95 405 296 290 290
query96 594 269 258 258
query97 3199 2975 3016 2975
query98 239 216 222 216
query99 1219 886 914 886
Total cold run time: 285190 ms
Total hot run time: 187209 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2116558999
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2116952422
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2116562505
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2117826721
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2103792507
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #34257:
URL: https://github.com/apache/doris/pull/34257#discussion_r1602988760
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java:
##########
@@ -576,12 +585,52 @@ public static long getIcebergRowCount(ExternalCatalog catalog, String dbName, St
}
public static String getFileFormat(Table table) {
- Snapshot snapshot = table.currentSnapshot();
- if (snapshot == null) {
- return TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
- } else {
- return snapshot.summary().getOrDefault(
- TableProperties.DEFAULT_FILE_FORMAT, TableProperties.DEFAULT_FILE_FORMAT_DEFAULT);
+ Map<String, String> properties = table.properties();
+ if (properties.containsKey(WRITE_FORMAT)) {
+ return properties.get(WRITE_FORMAT);
+ }
+ if (properties.containsKey(TableProperties.DEFAULT_FILE_FORMAT)) {
+ return properties.get(TableProperties.DEFAULT_FILE_FORMAT);
+ }
+ return TableProperties.DEFAULT_FILE_FORMAT_DEFAULT;
+ }
+
+ public static String getFileCompress(Table table) {
+ Map<String, String> properties = table.properties();
+ if (properties.containsKey(COMPRESSION_CODEC)) {
+ return properties.get(COMPRESSION_CODEC);
+ } else if (properties.containsKey(SPARK_SQL_COMPRESSION_CODEC)) {
+ return properties.get(SPARK_SQL_COMPRESSION_CODEC);
+ }
+ String fileFormat = getFileFormat(table);
+ if (fileFormat.equalsIgnoreCase("parquet")) {
+ return properties.getOrDefault(
+ TableProperties.PARQUET_COMPRESSION, TableProperties.PARQUET_COMPRESSION_DEFAULT_SINCE_1_4_0);
+ } else if (fileFormat.equalsIgnoreCase("orc")) {
+ return properties.getOrDefault(
+ TableProperties.ORC_COMPRESSION, TableProperties.ORC_COMPRESSION_DEFAULT);
+ } else if (fileFormat.equalsIgnoreCase("avro")) {
Review Comment:
We are not support avro yet
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##########
@@ -83,4 +90,17 @@ public long fetchRowCount() {
makeSureInitialized();
return IcebergUtils.getIcebergRowCount(getCatalog(), getDbName(), getName());
}
+
+ public Table getIcebergTable() {
Review Comment:
use `IcebergUtils.getIcebergTable` directly, it will get iceberg table from cache.
No need to add a table reference here.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -50,16 +49,25 @@
import java.util.Set;
import java.util.stream.Collectors;
-public class HiveTableSink extends DataSink {
+public class HiveTableSink extends BaseExternalTableDataSink {
- private HMSExternalTable targetTable;
- protected TDataSink tDataSink;
+ private final HMSExternalTable targetTable;
+ private static final HashSet<TFileFormatType> supportedTypes = new HashSet<TFileFormatType>() {{
+ add(TFileFormatType.FORMAT_ORC);
+ add(TFileFormatType.FORMAT_PARQUET);
+ add(TFileFormatType.FORMAT_CSV_PLAIN);
Review Comment:
FORMAT_CSV_PLAIN is not supported yet
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java:
##########
@@ -72,6 +76,9 @@ public static LogicalSink<? extends Plan> createUnboundTableSink(List<String> na
} else if (curCatalog instanceof HMSExternalCatalog) {
return new UnboundHiveTableSink<>(nameParts, colNames, hints, partitions,
dmlCommandType, Optional.empty(), Optional.empty(), plan);
+ } else if (curCatalog instanceof IcebergExternalCatalog) {
Review Comment:
I find another issue that above, if the curCatalog is HMSExternalCatalog, it will create a `UnboundHiveTableSink`. But in HMSCatalog, there may be hudi and iceberg table.
So I think we should check table's type too?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java:
##########
@@ -101,6 +108,9 @@ public static LogicalSink<? extends Plan> createUnboundTableSinkMaybeOverwrite(L
} else if (curCatalog instanceof HMSExternalCatalog && !isAutoDetectPartition) {
return new UnboundHiveTableSink<>(nameParts, colNames, hints, partitions,
dmlCommandType, Optional.empty(), Optional.empty(), plan);
+ } else if (curCatalog instanceof IcebergExternalCatalog && !isAutoDetectPartition) {
Review Comment:
ditto
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BaseExternalTableInsertExecutor.java:
##########
@@ -0,0 +1,159 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.commands.insert;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.common.UserException;
+import org.apache.doris.common.profile.SummaryProfile;
+import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.datasource.ExternalTable;
+import org.apache.doris.nereids.NereidsPlanner;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalSink;
+import org.apache.doris.planner.BaseExternalTableDataSink;
+import org.apache.doris.planner.DataSink;
+import org.apache.doris.planner.PlanFragment;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.QueryState;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.transaction.TransactionManager;
+import org.apache.doris.transaction.TransactionStatus;
+import org.apache.doris.transaction.TransactionType;
+
+import com.google.common.base.Strings;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Optional;
+
+/**
+ * Insert executor for base external table
+ */
+public abstract class BaseExternalTableInsertExecutor extends AbstractInsertExecutor {
+ private static final Logger LOG = LogManager.getLogger(BaseExternalTableInsertExecutor.class);
+ private static final long INVALID_TXN_ID = -1L;
+ protected long txnId = INVALID_TXN_ID;
+ protected TransactionStatus txnStatus = TransactionStatus.ABORTED;
+ protected final TransactionManager transactionManager;
+ protected final String catalogName;
+ protected Optional<SummaryProfile> summaryProfile = Optional.empty();
+
+ /**
+ * constructor
+ */
+ public BaseExternalTableInsertExecutor(ConnectContext ctx, ExternalTable table,
+ String labelName, NereidsPlanner planner,
+ Optional<InsertCommandContext> insertCtx,
+ boolean emptyInsert) {
+ super(ctx, table, labelName, planner, insertCtx, emptyInsert);
+ catalogName = table.getCatalog().getName();
+ transactionManager = table.getCatalog().getTransactionManager();
+
+ if (ConnectContext.get().getExecutor() != null) {
+ summaryProfile = Optional.of(ConnectContext.get().getExecutor().getSummaryProfile());
+ }
+ }
+
+ public long getTxnId() {
+ return txnId;
+ }
+
+ /**
+ * collect commit infos from BEs
+ */
+ public abstract void setCollectCommitInfoFunc();
+
+ @Override
+ public void beginTransaction() {
+ txnId = transactionManager.begin();
+ setCollectCommitInfoFunc();
+ }
+
+ /**
+ * At this time, FE has successfully collected all commit information from BEs.
+ * Before commit this txn, commit information need to be analyzed and processed.
+ */
+ protected abstract void doBeforeCommit() throws UserException;
Review Comment:
Better put all `abstract` method together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2081465695
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on code in PR #34257:
URL: https://github.com/apache/doris/pull/34257#discussion_r1603245455
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java:
##########
@@ -72,6 +76,9 @@ public static LogicalSink<? extends Plan> createUnboundTableSink(List<String> na
} else if (curCatalog instanceof HMSExternalCatalog) {
return new UnboundHiveTableSink<>(nameParts, colNames, hints, partitions,
dmlCommandType, Optional.empty(), Optional.empty(), plan);
+ } else if (curCatalog instanceof IcebergExternalCatalog) {
Review Comment:
This check will be done when it comes to logic to physics:
```
private Pair<HMSExternalDatabase, HMSExternalTable> bind(CascadesContext cascadesContext,
UnboundHiveTableSink<? extends Plan> sink) {
List<String> tableQualifier = RelationUtil.getQualifierName(cascadesContext.getConnectContext(),
sink.getNameParts());
Pair<DatabaseIf<?>, TableIf> pair = RelationUtil.getDbAndTable(tableQualifier,
cascadesContext.getConnectContext().getEnv());
if (pair.second instanceof HMSExternalTable) {
return Pair.of(((HMSExternalDatabase) pair.first), (HMSExternalTable) pair.second);
}
throw new AnalysisException("the target table of insert into is not a Hive table");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2114694119
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2117826783
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2115213493
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2095299981
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on code in PR #34257:
URL: https://github.com/apache/doris/pull/34257#discussion_r1603245455
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java:
##########
@@ -72,6 +76,9 @@ public static LogicalSink<? extends Plan> createUnboundTableSink(List<String> na
} else if (curCatalog instanceof HMSExternalCatalog) {
return new UnboundHiveTableSink<>(nameParts, colNames, hints, partitions,
dmlCommandType, Optional.empty(), Optional.empty(), plan);
+ } else if (curCatalog instanceof IcebergExternalCatalog) {
Review Comment:
This check will be done when it comes to logic to physics:
```
private Pair<HMSExternalDatabase, HMSExternalTable> bind(CascadesContext cascadesContext,
UnboundHiveTableSink<? extends Plan> sink) {
List<String> tableQualifier = RelationUtil.getQualifierName(cascadesContext.getConnectContext(),
sink.getNameParts());
Pair<DatabaseIf<?>, TableIf> pair = RelationUtil.getDbAndTable(tableQualifier,
cascadesContext.getConnectContext().getEnv());
if (pair.second instanceof HMSExternalTable) {
return Pair.of(((HMSExternalDatabase) pair.first), (HMSExternalTable) pair.second);
}
throw new AnalysisException("the target table of insert into is not a Hive table");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2115210669
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2115142582
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2114703559
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2095283088
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2095333207
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2095401634
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2081466935
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2114177129
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2116629023
<details>
<summary>ClickBench: <b>Total hot run time: 30.79 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86a406858bf5f0101df4a8e0be10483dbaa10b44, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.05
query3 0.23 0.05 0.05
query4 1.67 0.07 0.07
query5 0.50 0.50 0.51
query6 1.12 0.73 0.73
query7 0.02 0.02 0.01
query8 0.05 0.04 0.05
query9 0.54 0.49 0.48
query10 0.54 0.54 0.52
query11 0.16 0.11 0.11
query12 0.15 0.13 0.13
query13 0.60 0.59 0.60
query14 0.76 0.78 0.78
query15 0.82 0.82 0.81
query16 0.36 0.36 0.36
query17 1.04 0.99 1.01
query18 0.22 0.22 0.27
query19 1.79 1.79 1.79
query20 0.02 0.01 0.01
query21 15.73 0.65 0.65
query22 4.47 6.97 2.19
query23 18.28 1.44 1.22
query24 1.96 0.23 0.20
query25 0.14 0.08 0.08
query26 0.27 0.17 0.18
query27 0.08 0.09 0.08
query28 13.33 1.01 0.99
query29 13.81 3.32 3.27
query30 0.24 0.06 0.05
query31 2.92 0.40 0.38
query32 3.24 0.48 0.47
query33 2.76 2.86 2.81
query34 17.19 4.41 4.53
query35 4.48 4.51 4.65
query36 0.64 0.46 0.46
query37 0.17 0.15 0.16
query38 0.15 0.14 0.14
query39 0.04 0.03 0.04
query40 0.17 0.14 0.15
query41 0.10 0.04 0.04
query42 0.05 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 110.97 s
Total hot run time: 30.79 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](iceberg)support iceberg insert grammar [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #34257:
URL: https://github.com/apache/doris/pull/34257#issuecomment-2116946481
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org