You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wsjz (via GitHub)" <gi...@apache.org> on 2024/03/05 02:30:13 UTC
[PR] feature](insert)implement hive table sink plan [doris]
wsjz opened a new pull request, #31765:
URL: https://github.com/apache/doris/pull/31765
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "kaka11chen (via GitHub)" <gi...@apache.org>.
kaka11chen commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1519357612
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -314,13 +330,12 @@ struct THivePartition {
struct THiveTableSink {
1: optional string db_name
2: optional string table_name
- 3: optional list<string> data_column_names
- 4: optional list<string> partition_column_names
- 5: optional list<THivePartition> partitions
- 6: optional list<THiveBucket> buckets
- 7: optional PlanNodes.TFileFormatType file_format
- 8: optional THiveCompressionType compression_type
- 9: optional THiveLocationParams location
+ 3: optional list<THiveColumn> columns
+ 4: optional list<THivePartition> partitions
+ 5: optional THiveBucket bucket_info
+ 6: optional PlanNodes.TFileFormatType file_format
+ 7: optional THiveCompressionType compression_type
+ 8: optional THiveLocationParams location
Review Comment:
> Why here is a `THiveLocationParams`? There is already a `THiveLocationParams` in `THivePartition`
Table level and partition level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999330047
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1997317963
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2001916571
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2003008775
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999064056
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1996857098
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1525701440
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -299,10 +300,25 @@ struct THiveBucket {
}
enum THiveCompressionType {
- SNAPPY = 3,
- LZ4 = 4,
- ZLIB = 6,
- ZSTD = 7,
+ NO_COMPRESSION = 0,
Review Comment:
fix in https://github.com/apache/doris/pull/32274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999543066
<details>
<summary>TPC-H: <b>Total hot run time: 37911 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd6f8faa705566939ce26dbac7aece986308cfb5, data reload: false
------ Round 1 ----------------------------------
q1 17665 4153 4086 4086
q2 2020 154 142 142
q3 10608 1044 874 874
q4 7764 756 685 685
q5 7452 2663 2652 2652
q6 183 129 119 119
q7 1154 802 784 784
q8 9347 2007 1983 1983
q9 7153 6355 6383 6355
q10 8498 3479 3646 3479
q11 426 220 220 220
q12 689 285 291 285
q13 17907 2847 2831 2831
q14 265 241 246 241
q15 494 449 446 446
q16 493 396 386 386
q17 944 571 530 530
q18 7040 6406 6431 6406
q19 1697 1444 1371 1371
q20 543 281 276 276
q21 6066 3459 3559 3459
q22 349 331 301 301
Total cold run time: 108757 ms
Total hot run time: 37911 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4099 4039 4047 4039
q2 318 222 220 220
q3 2946 2806 2830 2806
q4 1827 1492 1512 1492
q5 5188 5231 5248 5231
q6 195 116 115 115
q7 2231 1812 1847 1812
q8 3136 3292 3294 3292
q9 8523 8513 8436 8436
q10 3606 3645 3655 3645
q11 535 446 443 443
q12 723 543 546 543
q13 16931 2868 2866 2866
q14 283 247 254 247
q15 490 442 441 441
q16 448 412 415 412
q17 1721 1479 1459 1459
q18 7365 7250 7145 7145
q19 1570 1481 1505 1481
q20 1910 1694 1699 1694
q21 4796 4662 4666 4662
q22 538 442 455 442
Total cold run time: 69379 ms
Total hot run time: 52923 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1526425439
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/PostgreSQLJdbcHMSCachedClient.java:
##########
@@ -172,15 +176,27 @@ public List<Partition> getPartitions(String dbName, String tblName, List<String>
private List<Partition> getPartitionsByNames(String dbName, String tblName, List<String> partitionNames) {
List<String> partitionNamesWithQuote = partitionNames.stream().map(partitionName -> "'" + partitionName + "'")
.collect(Collectors.toList());
- String partitionNamesString = Joiner.on(", ").join(partitionNamesWithQuote);
- String sql = String.format("SELECT \"PART_ID\", \"PARTITIONS\".\"CREATE_TIME\","
- + " \"PARTITIONS\".\"LAST_ACCESS_TIME\","
- + " \"PART_NAME\", \"PARTITIONS\".\"SD_ID\" FROM \"PARTITIONS\""
- + " join \"TBLS\" on \"TBLS\".\"TBL_ID\" = \"PARTITIONS\".\"TBL_ID\""
- + " join \"DBS\" on \"TBLS\".\"DB_ID\" = \"DBS\".\"DB_ID\""
- + " WHERE \"DBS\".\"NAME\" = '%s' AND \"TBLS\".\"TBL_NAME\"='%s'"
- + " AND \"PART_NAME\" in (%s);",
- dbName, tblName, partitionNamesString);
+ String sql;
+ if (partitionNamesWithQuote.isEmpty()) {
+ sql = String.format("SELECT \"PART_ID\", \"PARTITIONS\".\"CREATE_TIME\","
+ + " \"PARTITIONS\".\"LAST_ACCESS_TIME\","
+ + " \"PART_NAME\", \"PARTITIONS\".\"SD_ID\" FROM \"PARTITIONS\""
+ + " join \"TBLS\" on \"TBLS\".\"TBL_ID\" = \"PARTITIONS\".\"TBL_ID\""
+ + " join \"DBS\" on \"TBLS\".\"DB_ID\" = \"DBS\".\"DB_ID\""
+ + " WHERE \"DBS\".\"NAME\" = '%s' AND \"TBLS\".\"TBL_NAME\"='%s'"
+ + " AND \"PART_NAME\";",
Review Comment:
we need listPartitions when use postgresql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2001931907
<details>
<summary>TPC-H: <b>Total hot run time: 38347 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 050190a810e213856c8c67387db687583cc8f060, data reload: false
------ Round 1 ----------------------------------
q1 17649 4453 4122 4122
q2 2019 149 142 142
q3 10593 1073 908 908
q4 7772 821 727 727
q5 7463 2635 2685 2635
q6 187 127 124 124
q7 1203 812 809 809
q8 9319 1958 2027 1958
q9 7163 6474 6441 6441
q10 8528 3529 3658 3529
q11 427 226 226 226
q12 671 306 292 292
q13 18009 2849 2862 2849
q14 274 262 245 245
q15 500 465 452 452
q16 514 394 387 387
q17 966 541 549 541
q18 7302 6545 6418 6418
q19 2216 1415 1355 1355
q20 543 294 288 288
q21 6091 3619 3598 3598
q22 386 301 318 301
Total cold run time: 109795 ms
Total hot run time: 38347 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4113 4089 4086 4086
q2 315 227 216 216
q3 2946 2849 2828 2828
q4 1868 1545 1605 1545
q5 5240 5239 5240 5239
q6 192 116 118 116
q7 2217 1853 1830 1830
q8 3162 3296 3246 3246
q9 8538 8517 8554 8517
q10 3705 3636 3655 3636
q11 544 428 452 428
q12 707 542 539 539
q13 16918 2847 2888 2847
q14 271 251 251 251
q15 488 442 445 442
q16 448 408 415 408
q17 1733 1469 1453 1453
q18 7789 7194 7044 7044
q19 1632 1584 1525 1525
q20 1892 1689 1706 1689
q21 4689 4709 4789 4709
q22 551 457 484 457
Total cold run time: 69958 ms
Total hot run time: 53051 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1519134731
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys = targetTable.getRemoteTable().getPartitionKeys();
Review Comment:
```suggestion
List<FieldSchema> partitionKeys = targetTable.getPartitionColumnNames()
```
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys = targetTable.getRemoteTable().getPartitionKeys();
+
+ Set<String> nameToPartitions = new HashSet<>();
Review Comment:
```suggestion
Set<String> partitionNames = new HashSet<>();
```
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys = targetTable.getRemoteTable().getPartitionKeys();
+
+ Set<String> nameToPartitions = new HashSet<>();
+ for (FieldSchema partitionKey : partitionKeys) {
+ nameToPartitions.add(partitionKey.getName());
+ }
+ List<FieldSchema> hmsColumns = targetTable.getRemoteTable().getSd().getCols();
Review Comment:
Use schema in targetTable directly, no need to visit remote table
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -299,10 +300,25 @@ struct THiveBucket {
}
enum THiveCompressionType {
- SNAPPY = 3,
- LZ4 = 4,
- ZLIB = 6,
- ZSTD = 7,
+ NO_COMPRESSION = 0,
Review Comment:
Reuse the existing definition: TFileCompressType
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +81,128 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void init(List<Column> insertCols, List<Long> partitionIds) throws AnalysisException {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ List<FieldSchema> partitionKeys = targetTable.getRemoteTable().getPartitionKeys();
+
+ Set<String> nameToPartitions = new HashSet<>();
+ for (FieldSchema partitionKey : partitionKeys) {
+ nameToPartitions.add(partitionKey.getName());
+ }
+ List<FieldSchema> hmsColumns = targetTable.getRemoteTable().getSd().getCols();
+ Set<String> nameToColumns = new HashSet<>();
Review Comment:
```suggestion
Set<String> colNames = new HashSet<>();
```
##########
gensrc/thrift/DataSinks.thrift:
##########
@@ -314,13 +330,12 @@ struct THivePartition {
struct THiveTableSink {
1: optional string db_name
2: optional string table_name
- 3: optional list<string> data_column_names
- 4: optional list<string> partition_column_names
- 5: optional list<THivePartition> partitions
- 6: optional list<THiveBucket> buckets
- 7: optional PlanNodes.TFileFormatType file_format
- 8: optional THiveCompressionType compression_type
- 9: optional THiveLocationParams location
+ 3: optional list<THiveColumn> columns
+ 4: optional list<THivePartition> partitions
+ 5: optional THiveBucket bucket_info
+ 6: optional PlanNodes.TFileFormatType file_format
+ 7: optional THiveCompressionType compression_type
+ 8: optional THiveLocationParams location
Review Comment:
Why here is a `THiveLocationParams`?
There is already a `THiveLocationParams` in `THivePartition`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999479863
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1526440630
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java:
##########
@@ -345,13 +347,17 @@ private static Expression castValue(Expression value, DataType targetType) {
* get target table from names.
*/
public static TableIf getTargetTable(Plan plan, ConnectContext ctx) {
- if (!(plan instanceof UnboundTableSink)) {
- throw new AnalysisException("the root of plan should be UnboundTableSink"
- + " but it is " + plan.getType());
+ if (plan instanceof UnboundTableSink) {
Review Comment:
UnboundTableSink is too complex, so we should refactor to UnboundTableSink first.
Before the refactor, to simplify the problem，we should use 'if-else' to distinguish olap and hive for the moment.
。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2003008743
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999088034
<details>
<summary>TPC-H: <b>Total hot run time: 38573 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f4281393c816a44513eefb52868a5d51bf557e16, data reload: false
------ Round 1 ----------------------------------
q1 17627 4216 4130 4130
q2 2026 161 147 147
q3 10599 1079 917 917
q4 8121 785 730 730
q5 7486 2584 2640 2584
q6 195 124 125 124
q7 1181 827 815 815
q8 9336 2005 2002 2002
q9 7070 6422 6418 6418
q10 8479 3518 3666 3518
q11 431 231 221 221
q12 610 306 296 296
q13 17788 2886 2880 2880
q14 278 243 257 243
q15 499 453 452 452
q16 505 398 399 398
q17 983 606 561 561
q18 7464 6724 6542 6542
q19 2592 1414 1454 1414
q20 554 295 284 284
q21 6531 3610 3582 3582
q22 393 315 321 315
Total cold run time: 110748 ms
Total hot run time: 38573 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4153 4123 4113 4113
q2 322 227 218 218
q3 3045 2912 2916 2912
q4 1879 1613 1582 1582
q5 5247 5293 5280 5280
q6 201 116 120 116
q7 2257 1869 1840 1840
q8 3180 3331 3318 3318
q9 8625 8530 8580 8530
q10 3773 3706 3776 3706
q11 543 442 438 438
q12 729 556 530 530
q13 16909 2876 2854 2854
q14 297 243 251 243
q15 490 447 447 447
q16 458 428 424 424
q17 1761 1491 1478 1478
q18 7561 7296 7195 7195
q19 1632 1541 1529 1529
q20 1924 1773 1750 1750
q21 4887 4732 4871 4732
q22 549 469 458 458
Total cold run time: 70422 ms
Total hot run time: 53693 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999443053
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1526076032
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java:
##########
@@ -878,6 +878,11 @@ public Map<Long, PartitionItem> getAndCopyPartitionItems() {
return hivePartitionValues.getIdToPartitionItem();
}
+ public Long getPartitionId(String partName) {
Review Comment:
Not used?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/UnboundTableSinkCreator.java:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.analyzer;
+
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.datasource.hive.HMSExternalCatalog;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.commands.info.DMLCommandType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSink;
+import org.apache.doris.nereids.util.RelationUtil;
+import org.apache.doris.qe.ConnectContext;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Create unbound table sink
+ */
+public class UnboundTableSinkCreator {
+
+ /**
+ * create unbound sink
+ */
+ public static LogicalSink<? extends Plan> createUnboundTableSink(List<String> nameParts,
+ List<String> colNames, List<String> hints, List<String> partitions, Plan query)
+ throws UserException {
+ String catalogName = RelationUtil.getQualifierName(ConnectContext.get(), nameParts).get(0);
+ CatalogIf<?> curCatalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+ if (curCatalog instanceof InternalCatalog) {
+ return new UnboundTableSink<>(nameParts, colNames, hints, partitions, query);
+ } else if (curCatalog instanceof HMSExternalCatalog) {
+ return new UnboundHiveTableSink<>(nameParts, colNames, hints, partitions, query);
+ }
+ throw new UserException("Load data to " + curCatalog.getClass().getSimpleName() + " is not supported.");
+ }
+
+ /**
Review Comment:
Add comment to explain the difference of 2 methods
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +85,132 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void bindDataSink(List<Column> insertCols, Optional<InsertCommandContext> insertCtx) {
Review Comment:
Add some comment to explain the whole logic
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +85,132 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void bindDataSink(List<Column> insertCols, Optional<InsertCommandContext> insertCtx) {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ Set<String> partNames = new HashSet<>(targetTable.getPartitionColumnNames());
+ Set<String> colNames = targetTable.getColumns()
+ .stream().map(Column::getName)
+ .collect(Collectors.toSet());
+ colNames.removeAll(partNames);
+ List<THiveColumn> targetColumns = new ArrayList<>();
+ for (Column col : insertCols) {
+ if (partNames.contains(col.getName())) {
+ THiveColumn tHiveColumn = new THiveColumn();
+ tHiveColumn.setName(col.getName());
+ tHiveColumn.setDataType(col.getType().toThrift());
+ tHiveColumn.setColumnType(THiveColumnType.PARTITION_KEY);
+ targetColumns.add(tHiveColumn);
+ } else if (colNames.contains(col.getName())) {
+ THiveColumn tHiveColumn = new THiveColumn();
+ tHiveColumn.setName(col.getName());
+ tHiveColumn.setDataType(col.getType().toThrift());
+ tHiveColumn.setColumnType(THiveColumnType.REGULAR);
+ targetColumns.add(tHiveColumn);
+ }
+ }
+ tSink.setColumns(targetColumns);
+
+ setPartitionValues(tSink);
+
+ StorageDescriptor sd = targetTable.getRemoteTable().getSd();
+ THiveBucket bucketInfo = new THiveBucket();
+ bucketInfo.setBucketedBy(sd.getBucketCols());
+ bucketInfo.setBucketCount(sd.getNumBuckets());
+ tSink.setBucketInfo(bucketInfo);
+
+ TFileFormatType formatType = getFileFormatType(sd);
+ tSink.setFileFormat(formatType);
+ setCompressType(tSink, formatType);
+
+ THiveLocationParams locationParams = new THiveLocationParams();
+ String location = sd.getLocation();
+
+ String writeTempPath = createTempPath(location);
+ locationParams.setWritePath(writeTempPath);
+ locationParams.setTargetPath(location);
+ locationParams.setFileType(LocationPath.getTFileTypeForBE(location));
+ tSink.setLocation(locationParams);
+
+ tSink.setHadoopConfig(targetTable.getHadoopProperties());
+
+ if (insertCtx.isPresent()) {
+ HiveInsertCommandContext context = (HiveInsertCommandContext) insertCtx.get();
+ tSink.setOverwrite(context.isOverwrite());
+ }
+ tDataSink = new TDataSink(getDataSinkType());
+ tDataSink.setHiveTableSink(tSink);
+ }
+
+ private String createTempPath(String location) {
+ String user = ConnectContext.get().getUserIdentity().getUser();
+ return location + "/.doris_staging/" + user + "/" + UUID.randomUUID().toString().replace("-", "");
+ }
+
+ private void setCompressType(THiveTableSink tSink, TFileFormatType formatType) {
+ String compressType;
+ switch (formatType) {
+ case FORMAT_ORC:
+ compressType = targetTable.getRemoteTable().getParameters().get("orc.compress");
+ break;
+ case FORMAT_PARQUET:
+ compressType = targetTable.getRemoteTable().getParameters().get("parquet.compression");
+ break;
+ default:
+ compressType = "uncompressed";
+ break;
+ }
+
+ if ("snappy".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.SNAPPYBLOCK);
+ } else if ("lz4".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.LZ4BLOCK);
+ } else if ("lzo".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.LZO);
+ } else if ("zlib".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.ZLIB);
+ } else if ("zstd".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.ZSTD);
+ } else {
+ tSink.setCompressionType(TFileCompressType.PLAIN);
Review Comment:
Should throw exception for unsupported compress type
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalSink.java:
##########
@@ -54,4 +54,8 @@ public List<Slot> computeOutput() {
.map(NamedExpression::toSlot)
.collect(ImmutableList.toImmutableList());
}
+
+ public PhysicalProperties getRequirePhysicalProperties() {
+ return PhysicalProperties.ANY;
Review Comment:
Double check whether `PhysicalProperties.ANY` is the default value for PhysicalSink.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java:
##########
@@ -156,20 +160,40 @@ public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
*/
private void insertInto(ConnectContext ctx, StmtExecutor executor, List<String> tempPartitionNames)
throws Exception {
- UnboundTableSink<?> sink = (UnboundTableSink<?>) logicalQuery;
- UnboundTableSink<?> copySink = new UnboundTableSink<>(
- sink.getNameParts(),
- sink.getColNames(),
- sink.getHints(),
- true,
- tempPartitionNames,
- sink.isPartialUpdate(),
- sink.getDMLCommandType(),
- (LogicalPlan) (sink.child(0)));
- // for overwrite situation, we disable auto create partition.
- OlapInsertCommandContext insertCtx = new OlapInsertCommandContext();
- insertCtx.setAllowAutoPartition(false);
- InsertIntoTableCommand insertCommand = new InsertIntoTableCommand(copySink, labelName, Optional.of(insertCtx));
+ UnboundLogicalSink<?> copySink;
+ InsertCommandContext insertCtx;
+ if (logicalQuery instanceof UnboundTableSink) {
+ UnboundTableSink<?> sink = (UnboundTableSink<?>) logicalQuery;
+ copySink = (UnboundLogicalSink<?>) UnboundTableSinkCreator.createUnboundTableSink(
+ sink.getNameParts(),
+ sink.getColNames(),
+ sink.getHints(),
+ true,
+ tempPartitionNames,
+ sink.isPartialUpdate(),
+ sink.getDMLCommandType(),
+ (LogicalPlan) (sink.child(0)));
+ // for overwrite situation, we disable auto create partition.
+ insertCtx = new OlapInsertCommandContext();
+ ((OlapInsertCommandContext) insertCtx).setAllowAutoPartition(false);
+ } else if (logicalQuery instanceof UnboundHiveTableSink) {
+ UnboundHiveTableSink<?> sink = (UnboundHiveTableSink<?>) logicalQuery;
+ copySink = (UnboundLogicalSink<?>) UnboundTableSinkCreator.createUnboundTableSink(
+ sink.getNameParts(),
+ sink.getColNames(),
+ sink.getHints(),
+ false,
+ sink.getPartitions(),
+ false,
+ sink.getDMLCommandType(),
+ (LogicalPlan) (sink.child(0)));
+ insertCtx = new HiveInsertCommandContext();
+ ((HiveInsertCommandContext) insertCtx).setOverwrite(false);
+ } else {
+ throw new RuntimeException("Current catalog has not supported insert overwrite yet.");
Review Comment:
```suggestion
throw new RuntimeException("Current catalog does not support insert overwrite.");
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/PostgreSQLJdbcHMSCachedClient.java:
##########
@@ -172,15 +176,27 @@ public List<Partition> getPartitions(String dbName, String tblName, List<String>
private List<Partition> getPartitionsByNames(String dbName, String tblName, List<String> partitionNames) {
List<String> partitionNamesWithQuote = partitionNames.stream().map(partitionName -> "'" + partitionName + "'")
.collect(Collectors.toList());
- String partitionNamesString = Joiner.on(", ").join(partitionNamesWithQuote);
- String sql = String.format("SELECT \"PART_ID\", \"PARTITIONS\".\"CREATE_TIME\","
- + " \"PARTITIONS\".\"LAST_ACCESS_TIME\","
- + " \"PART_NAME\", \"PARTITIONS\".\"SD_ID\" FROM \"PARTITIONS\""
- + " join \"TBLS\" on \"TBLS\".\"TBL_ID\" = \"PARTITIONS\".\"TBL_ID\""
- + " join \"DBS\" on \"TBLS\".\"DB_ID\" = \"DBS\".\"DB_ID\""
- + " WHERE \"DBS\".\"NAME\" = '%s' AND \"TBLS\".\"TBL_NAME\"='%s'"
- + " AND \"PART_NAME\" in (%s);",
- dbName, tblName, partitionNamesString);
+ String sql;
+ if (partitionNamesWithQuote.isEmpty()) {
+ sql = String.format("SELECT \"PART_ID\", \"PARTITIONS\".\"CREATE_TIME\","
+ + " \"PARTITIONS\".\"LAST_ACCESS_TIME\","
+ + " \"PART_NAME\", \"PARTITIONS\".\"SD_ID\" FROM \"PARTITIONS\""
+ + " join \"TBLS\" on \"TBLS\".\"TBL_ID\" = \"PARTITIONS\".\"TBL_ID\""
+ + " join \"DBS\" on \"TBLS\".\"DB_ID\" = \"DBS\".\"DB_ID\""
+ + " WHERE \"DBS\".\"NAME\" = '%s' AND \"TBLS\".\"TBL_NAME\"='%s'"
+ + " AND \"PART_NAME\";",
Review Comment:
The last is `AND PARTNAME`?
And why changing this? This is unrelated to your feature
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalHiveTableSink.java:
##########
@@ -100,20 +118,50 @@ public List<? extends Expression> getExpressions() {
@Override
public Plan withGroupExpression(Optional<GroupExpression> groupExpression) {
- return new PhysicalHiveTableSink<>(database, targetTable, cols, partitionIds, outputExprs,
- groupExpression, getLogicalProperties(), child());
+ return new PhysicalHiveTableSink<>(database, targetTable, cols, outputExprs,
+ groupExpression, getLogicalProperties(), child(), hivePartitionKeys);
}
@Override
public Plan withGroupExprLogicalPropChildren(Optional<GroupExpression> groupExpression,
Optional<LogicalProperties> logicalProperties, List<Plan> children) {
- return new PhysicalHiveTableSink<>(database, targetTable, cols, partitionIds, outputExprs,
- groupExpression, logicalProperties.get(), children.get(0));
+ return new PhysicalHiveTableSink<>(database, targetTable, cols, outputExprs,
+ groupExpression, logicalProperties.get(), children.get(0), hivePartitionKeys);
}
@Override
public PhysicalPlan withPhysicalPropertiesAndStats(PhysicalProperties physicalProperties, Statistics statistics) {
- return new PhysicalHiveTableSink<>(database, targetTable, cols, partitionIds, outputExprs,
- groupExpression, getLogicalProperties(), physicalProperties, statistics, child());
+ return new PhysicalHiveTableSink<>(database, targetTable, cols, outputExprs,
+ groupExpression, getLogicalProperties(), physicalProperties, statistics, child(), hivePartitionKeys);
+ }
+
+ /**
+ * get output physical properties
+ */
+ @Override
+ public PhysicalProperties getRequirePhysicalProperties() {
+ Set<String> hivePartitionKeys = targetTable.getRemoteTable()
+ .getPartitionKeys().stream()
+ .map(FieldSchema::getName)
+ .collect(Collectors.toSet());
+ if (!hivePartitionKeys.isEmpty()) {
+ List<Integer> columnIdx = new ArrayList<>();
+ List<Column> fullSchema = targetTable.getFullSchema();
+ for (int i = 0; i < fullSchema.size(); i++) {
+ Column column = fullSchema.get(i);
+ if (hivePartitionKeys.contains(column.getName())) {
+ columnIdx.add(i);
+ }
+ }
+ DistributionSpecTableSinkHashPartitioned shuffleInfo =
+ (DistributionSpecTableSinkHashPartitioned) PhysicalProperties.SINK_HASH_PARTITIONED
Review Comment:
`PhysicalProperties.SINK_HASH_PARTITIONED` is a singleton,
can we modify it?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java:
##########
@@ -345,13 +347,17 @@ private static Expression castValue(Expression value, DataType targetType) {
* get target table from names.
*/
public static TableIf getTargetTable(Plan plan, ConnectContext ctx) {
- if (!(plan instanceof UnboundTableSink)) {
- throw new AnalysisException("the root of plan should be UnboundTableSink"
- + " but it is " + plan.getType());
+ if (plan instanceof UnboundTableSink) {
Review Comment:
What is the difference between this 2 branch?
We need to try best to avoid `if else` for different type of table
##########
fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java:
##########
@@ -59,9 +85,132 @@ public DataPartition getOutputPartition() {
return DataPartition.RANDOM;
}
- public void init() {
+ public void bindDataSink(List<Column> insertCols, Optional<InsertCommandContext> insertCtx) {
+ THiveTableSink tSink = new THiveTableSink();
+ tSink.setDbName(targetTable.getDbName());
+ tSink.setTableName(targetTable.getName());
+ Set<String> partNames = new HashSet<>(targetTable.getPartitionColumnNames());
+ Set<String> colNames = targetTable.getColumns()
+ .stream().map(Column::getName)
+ .collect(Collectors.toSet());
+ colNames.removeAll(partNames);
+ List<THiveColumn> targetColumns = new ArrayList<>();
+ for (Column col : insertCols) {
+ if (partNames.contains(col.getName())) {
+ THiveColumn tHiveColumn = new THiveColumn();
+ tHiveColumn.setName(col.getName());
+ tHiveColumn.setDataType(col.getType().toThrift());
+ tHiveColumn.setColumnType(THiveColumnType.PARTITION_KEY);
+ targetColumns.add(tHiveColumn);
+ } else if (colNames.contains(col.getName())) {
+ THiveColumn tHiveColumn = new THiveColumn();
+ tHiveColumn.setName(col.getName());
+ tHiveColumn.setDataType(col.getType().toThrift());
+ tHiveColumn.setColumnType(THiveColumnType.REGULAR);
+ targetColumns.add(tHiveColumn);
+ }
+ }
+ tSink.setColumns(targetColumns);
+
+ setPartitionValues(tSink);
+
+ StorageDescriptor sd = targetTable.getRemoteTable().getSd();
+ THiveBucket bucketInfo = new THiveBucket();
+ bucketInfo.setBucketedBy(sd.getBucketCols());
+ bucketInfo.setBucketCount(sd.getNumBuckets());
+ tSink.setBucketInfo(bucketInfo);
+
+ TFileFormatType formatType = getFileFormatType(sd);
+ tSink.setFileFormat(formatType);
+ setCompressType(tSink, formatType);
+
+ THiveLocationParams locationParams = new THiveLocationParams();
+ String location = sd.getLocation();
+
+ String writeTempPath = createTempPath(location);
+ locationParams.setWritePath(writeTempPath);
+ locationParams.setTargetPath(location);
+ locationParams.setFileType(LocationPath.getTFileTypeForBE(location));
+ tSink.setLocation(locationParams);
+
+ tSink.setHadoopConfig(targetTable.getHadoopProperties());
+
+ if (insertCtx.isPresent()) {
+ HiveInsertCommandContext context = (HiveInsertCommandContext) insertCtx.get();
+ tSink.setOverwrite(context.isOverwrite());
+ }
+ tDataSink = new TDataSink(getDataSinkType());
+ tDataSink.setHiveTableSink(tSink);
+ }
+
+ private String createTempPath(String location) {
+ String user = ConnectContext.get().getUserIdentity().getUser();
+ return location + "/.doris_staging/" + user + "/" + UUID.randomUUID().toString().replace("-", "");
+ }
+
+ private void setCompressType(THiveTableSink tSink, TFileFormatType formatType) {
+ String compressType;
+ switch (formatType) {
+ case FORMAT_ORC:
+ compressType = targetTable.getRemoteTable().getParameters().get("orc.compress");
+ break;
+ case FORMAT_PARQUET:
+ compressType = targetTable.getRemoteTable().getParameters().get("parquet.compression");
+ break;
+ default:
+ compressType = "uncompressed";
+ break;
+ }
+
+ if ("snappy".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.SNAPPYBLOCK);
+ } else if ("lz4".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.LZ4BLOCK);
+ } else if ("lzo".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.LZO);
+ } else if ("zlib".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.ZLIB);
+ } else if ("zstd".equalsIgnoreCase(compressType)) {
+ tSink.setCompressionType(TFileCompressType.ZSTD);
+ } else {
+ tSink.setCompressionType(TFileCompressType.PLAIN);
+ }
+ }
+
+ private void setPartitionValues(THiveTableSink tSink) {
+ List<THivePartition> partitions = new ArrayList<>();
+ List<org.apache.hadoop.hive.metastore.api.Partition> hivePartitions =
+ ((HMSExternalCatalog) targetTable.getCatalog())
+ .getClient().listPartitions(targetTable.getDbName(), targetTable.getName());
+ for (org.apache.hadoop.hive.metastore.api.Partition partition : hivePartitions) {
+ THivePartition hivePartition = new THivePartition();
+ StorageDescriptor sd = partition.getSd();
+ hivePartition.setFileFormat(getFileFormatType(sd));
+
+ hivePartition.setValues(partition.getValues());
+ THiveLocationParams locationParams = new THiveLocationParams();
+ String location = sd.getLocation();
+ // pass the same of write path and target path to partition
+ locationParams.setWritePath(location);
+ locationParams.setTargetPath(location);
+ locationParams.setFileType(LocationPath.getTFileTypeForBE(location));
+ hivePartition.setLocation(locationParams);
+ partitions.add(hivePartition);
+ }
+ tSink.setPartitions(partitions);
+ }
+
+ private TFileFormatType getFileFormatType(StorageDescriptor sd) {
+ TFileFormatType fileFormatType;
+ if (sd.getInputFormat().toLowerCase().contains("orc")) {
+ fileFormatType = TFileFormatType.FORMAT_ORC;
+ } else {
Review Comment:
else if parquet xxxx
else throw exception
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #31765:
URL: https://github.com/apache/doris/pull/31765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on code in PR #31765:
URL: https://github.com/apache/doris/pull/31765#discussion_r1527074698
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalSink.java:
##########
@@ -54,4 +54,8 @@ public List<Slot> computeOutput() {
.map(NamedExpression::toSlot)
.collect(ImmutableList.toImmutableList());
}
+
+ public PhysicalProperties getRequirePhysicalProperties() {
+ return PhysicalProperties.ANY;
Review Comment:
OK，we should use PhysicalProperties.GATHER and this method now is only called by hive and olap sink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999059821
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1998855099
issue: https://github.com/apache/doris/issues/31442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1999387999
<details>
<summary>TPC-H: <b>Total hot run time: 38041 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3bca495db7393c6cca1ecaec65bde6f5bdfcf81, data reload: false
------ Round 1 ----------------------------------
q1 17714 4202 4089 4089
q2 2021 151 142 142
q3 10597 1050 862 862
q4 7775 752 714 714
q5 7455 2719 2746 2719
q6 186 120 120 120
q7 1130 794 806 794
q8 9336 1966 1992 1966
q9 7094 6423 6403 6403
q10 8547 3524 3577 3524
q11 426 228 210 210
q12 767 303 291 291
q13 18094 2851 2858 2851
q14 268 254 256 254
q15 495 453 437 437
q16 503 390 394 390
q17 941 504 632 504
q18 7010 6392 6367 6367
q19 3578 1414 1398 1398
q20 537 277 265 265
q21 6022 3610 3447 3447
q22 349 294 314 294
Total cold run time: 110845 ms
Total hot run time: 38041 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4076 4093 4051 4051
q2 328 225 235 225
q3 2957 2862 2802 2802
q4 1861 1551 1559 1551
q5 5224 5238 5250 5238
q6 192 115 117 115
q7 2245 1828 1830 1828
q8 3122 3276 3280 3276
q9 8573 8557 8531 8531
q10 3701 3697 3676 3676
q11 550 437 437 437
q12 703 557 547 547
q13 16918 2885 2839 2839
q14 271 236 263 236
q15 473 441 443 441
q16 464 407 409 407
q17 1722 1459 1446 1446
q18 7420 7163 7130 7130
q19 4299 1537 1537 1537
q20 1901 1723 1703 1703
q21 4744 4662 4743 4662
q22 542 456 452 452
Total cold run time: 72286 ms
Total hot run time: 53130 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2002503804
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2002809679
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1977835984
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1998842453
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1998848649
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] feature](insert)implement hive table sink plan [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-1997297103
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](insert)implement hive table sink plan [doris]
Posted by "wsjz (via GitHub)" <gi...@apache.org>.
wsjz commented on PR #31765:
URL: https://github.com/apache/doris/pull/31765#issuecomment-2002743124
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org