You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "amorynan (via GitHub)" <gi...@apache.org> on 2024/01/19 10:12:12 UTC
[PR] [Feature](explode) support explode map type [doris]
amorynan opened a new pull request, #30151:
URL: https://github.com/apache/doris/pull/30151
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1956028056
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1916587493
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 1be87bdad54ff870d146fa9f596f94b21732d696 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495540175
##########
be/src/vec/exprs/table_function/vexplode_map.h:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+
+#include "common/status.h"
+#include "vec/columns/column_map.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_map.h"
+#include "vec/exprs/table_function/table_function.h"
+#include "vec/functions/array/function_array_utils.h"
+
+namespace doris {
+namespace vectorized {
+class Block;
+} // namespace vectorized
+} // namespace doris
+
+namespace doris::vectorized {
+
+class ColumnMapExecutionData {
+public:
+ void reset() {
+ map_nullmap_data = nullptr;
+ map_col = nullptr;
+ offsets_ptr = nullptr;
+ }
+
+public:
Review Comment:
here seems should be `private` ?
and shoud `_map_nullmap_data`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495492338
##########
regression-test/suites/nereids_p0/sql_functions/table_function/explode_map.groovy:
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("explode_map") {
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+
+ sql """ DROP TABLE IF EXISTS sdu """
+ sql """
+ CREATE TABLE IF NOT EXISTS `sdu`(
+ `id` INT NULL,
+ `name` TEXT NULL,
+ `score` MAP<TEXT,INT> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ // insert values
+ sql """ insert into sdu values (0, "zhangsan", {"Chinese":"80","Math":"60","English":"90"}); """
+ sql """ insert into sdu values (1, "lisi", {"null":null}); """
+ sql """ insert into sdu values (2, "wangwu", {"Chinese":"88","Math":"90","English":"96"}); """
+ sql """ insert into sdu values (3, "lisi2", {null:null}); """
+ sql """ insert into sdu values (4, "amory", NULL); """
+
+ qt_sql """ select * from sdu order by id; """
+ qt_explode_sql """ select name, k,v from sdu lateral view explode_map(score) tmp as k,v;"""
Review Comment:
add order by
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1955853111
TeamCity be ut coverage result:
Function Coverage: 35.70% (8558/23969)
Line Coverage: 27.59% (69415/251627)
Region Coverage: 26.73% (36025/134786)
Branch Coverage: 23.54% (18423/78258)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5ab4b4497e353b41d01762c1d838cf005bbf835a_5ab4b4497e353b41d01762c1d838cf005bbf835a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495538539
##########
be/src/vec/exprs/table_function/vexplode_map.h:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+
+#include "common/status.h"
+#include "vec/columns/column_map.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_map.h"
+#include "vec/exprs/table_function/table_function.h"
+#include "vec/functions/array/function_array_utils.h"
+
+namespace doris {
+namespace vectorized {
Review Comment:
shoud change like `namespace doris::vectorized {`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1953629009
<details>
<summary>TPC-H: <b>Total hot run time: 41349 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d09e6481faba217bcd588c792b6ffd773eeeefb, data reload: false
------ Round 1 ----------------------------------
q1 17610 4984 4840 4840
q2 2046 137 131 131
q3 10616 1034 1013 1013
q4 4747 950 976 950
q5 7744 3243 3264 3243
q6 194 131 130 130
q7 1263 783 758 758
q8 9369 2109 2089 2089
q9 7583 6679 6687 6679
q10 8313 2631 2633 2631
q11 416 209 210 209
q12 725 338 336 336
q13 18006 3634 3677 3634
q14 296 258 253 253
q15 607 495 506 495
q16 475 413 421 413
q17 928 869 852 852
q18 7549 6614 6523 6523
q19 1647 1501 1502 1501
q20 590 341 348 341
q21 6888 4041 3994 3994
q22 880 346 334 334
Total cold run time: 108492 ms
Total hot run time: 41349 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4830 4836 4833 4833
q2 292 181 187 181
q3 3591 3556 3587 3556
q4 2535 2532 2517 2517
q5 5757 5723 5736 5723
q6 213 127 127 127
q7 2234 1639 1631 1631
q8 2992 3077 3075 3075
q9 8642 8674 8677 8674
q10 6771 4228 4234 4228
q11 523 357 358 357
q12 760 546 561 546
q13 4365 3394 3400 3394
q14 255 238 237 237
q15 630 507 496 496
q16 464 414 421 414
q17 1631 1580 1583 1580
q18 8296 7639 7568 7568
q19 1635 1644 1641 1641
q20 2103 1803 1826 1803
q21 6535 6173 6188 6173
q22 572 499 502 499
Total cold run time: 65626 ms
Total hot run time: 59253 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495544609
##########
be/src/vec/exprs/table_function/vexplode_map.h:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+
+#include "common/status.h"
+#include "vec/columns/column_map.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_map.h"
+#include "vec/exprs/table_function/table_function.h"
+#include "vec/functions/array/function_array_utils.h"
+
+namespace doris {
+namespace vectorized {
+class Block;
+} // namespace vectorized
+} // namespace doris
+
+namespace doris::vectorized {
+
+class ColumnMapExecutionData {
+public:
+ void reset() {
+ map_nullmap_data = nullptr;
+ map_col = nullptr;
+ offsets_ptr = nullptr;
+ }
+
+public:
Review Comment:
maybe not use _map_nullmap_data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1955840639
<details>
<summary>TPC-H: <b>Total hot run time: 41572 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5ab4b4497e353b41d01762c1d838cf005bbf835a, data reload: false
------ Round 1 ----------------------------------
q1 17625 4973 4869 4869
q2 2049 146 133 133
q3 10667 1036 1038 1036
q4 4752 979 1003 979
q5 7722 3248 3300 3248
q6 194 138 131 131
q7 1262 786 767 767
q8 9532 2100 2087 2087
q9 7612 6692 6694 6692
q10 8381 2648 2640 2640
q11 419 204 222 204
q12 826 342 336 336
q13 17996 3694 3680 3680
q14 298 259 266 259
q15 615 508 494 494
q16 539 403 431 403
q17 945 892 860 860
q18 7135 6732 6644 6644
q19 1546 1489 1482 1482
q20 577 352 349 349
q21 6627 3987 3948 3948
q22 402 345 331 331
Total cold run time: 107721 ms
Total hot run time: 41572 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4830 4785 4801 4785
q2 299 180 184 180
q3 3619 3589 3589 3589
q4 2539 2509 2514 2509
q5 5792 5754 5760 5754
q6 214 125 126 125
q7 2243 1620 1647 1620
q8 2991 3103 3079 3079
q9 8724 8724 8728 8724
q10 4151 4239 4231 4231
q11 505 372 354 354
q12 773 542 550 542
q13 4192 3431 3411 3411
q14 257 238 234 234
q15 638 499 491 491
q16 476 427 447 427
q17 1663 1599 1617 1599
q18 8374 7702 7591 7591
q19 1642 1641 1641 1641
q20 2119 1835 1819 1819
q21 6546 6159 6102 6102
q22 587 515 520 515
Total cold run time: 63174 ms
Total hot run time: 59322 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30151:
URL: https://github.com/apache/doris/pull/30151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958653653
TeamCity be ut coverage result:
Function Coverage: 35.73% (8542/23905)
Line Coverage: 27.55% (69316/251614)
Region Coverage: 26.70% (35956/134667)
Branch Coverage: 23.50% (18378/78208)
Coverage Report: http://coverage.selectdb-in.cc/coverage/59d4ae2d3358788a223fce971e8cc2983ad56620_59d4ae2d3358788a223fce971e8cc2983ad56620/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958994759
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1955995903
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1953607589
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1916552577
<details>
<summary>TPC-H: <b>Total hot run time: 37325 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1be87bdad54ff870d146fa9f596f94b21732d696, data reload: false
------ Round 1 ----------------------------------
q1 17828 4767 4493 4493
q2 2045 138 130 130
q3 10680 903 897 897
q4 4677 745 811 745
q5 7990 2805 2827 2805
q6 186 123 123 123
q7 1137 729 706 706
q8 9299 1995 2017 1995
q9 9294 6555 6483 6483
q10 8758 2577 2584 2577
q11 412 213 209 209
q12 706 281 279 279
q13 18788 3387 3344 3344
q14 278 297 249 249
q15 538 499 480 480
q16 815 470 416 416
q17 943 571 509 509
q18 6785 6089 5931 5931
q19 1570 1444 1359 1359
q20 604 344 325 325
q21 6392 3069 2975 2975
q22 787 314 295 295
Total cold run time: 110512 ms
Total hot run time: 37325 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4487 4365 4399 4365
q2 323 231 226 226
q3 2961 2819 2802 2802
q4 1825 1671 1603 1603
q5 5193 5278 5232 5232
q6 191 116 116 116
q7 2108 1728 1827 1728
q8 3100 3232 3257 3232
q9 8330 8295 8230 8230
q10 5757 3526 3577 3526
q11 534 459 439 439
q12 774 591 557 557
q13 12071 3096 3101 3096
q14 279 246 254 246
q15 526 486 488 486
q16 509 490 467 467
q17 1839 1646 1707 1646
q18 7940 7755 7530 7530
q19 8333 1525 1524 1524
q20 2163 1897 1907 1897
q21 6641 4722 4700 4700
q22 548 456 513 456
Total cold run time: 76432 ms
Total hot run time: 54104 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1497010916
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java:
##########
@@ -1929,6 +1930,15 @@ private void initTableFunction() {
addTableFunctionWithCombinator(EXPLODE, Type.WILDCARD_DECIMAL, Function.NullableMode.ALWAYS_NULLABLE,
Lists.newArrayList(new ArrayType(Type.WILDCARD_DECIMAL)), false,
"_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
+
+ initTableFunctionListWithCombinator(EXPLODE_MAP);
+ for (Type keyType : Type.getMapSubTypes()) {
+ for (Type valueType : Type.getMapSubTypes()) {
+ addTableFunctionWithCombinator(EXPLODE_MAP, new StructType(keyType, valueType), Function.NullableMode.ALWAYS_NULLABLE,
+ Lists.newArrayList(new MapType(keyType, valueType)), false,
+ "_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
Review Comment:
why still need function symbol here?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGenerate.java:
##########
@@ -47,17 +47,25 @@ public class LogicalGenerate<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD
private final List<Function> generators;
private final List<Slot> generatorOutput;
+ // mapping with function.
+ private final List<List<String>> expandColumnAlias;
public LogicalGenerate(List<Function> generators, List<Slot> generatorOutput, CHILD_TYPE child) {
- this(generators, generatorOutput, Optional.empty(), Optional.empty(), child);
+ this(generators, generatorOutput, Lists.newArrayList(), Optional.empty(), Optional.empty(), child);
Review Comment:
```suggestion
this(generators, generatorOutput, ImmutableList.of(), Optional.empty(), Optional.empty(), child);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java:
##########
@@ -624,8 +631,34 @@ protected boolean condition(Rule rule, Plan plan) {
Slot boundSlot = new SlotReference(slot.getNameParts().get(1), generator.getDataType(),
generator.nullable(), ImmutableList.of(slot.getNameParts().get(0)));
slotBuilder.add(boundSlot);
+ // the boundSlot may has two situation:
+ // 1. the expandColumnsAlias is not empty, we should use make boundSlot expand to multi alias
+ // 2. the expandColumnsAlias is empty, we should use origin boundSlot
+ if (generate.getExpandColumnAlias() != null
+ && !CollectionUtils.isEmpty(generate.getExpandColumnAlias().get(i))) {
+ // if the alias is not empty, we should bind it with struct_element as child expr with alias
+ // struct_element(#expand_col#k, #k) as #k
+ // struct_element(#expand_col#v, #v) as #v
+ List<StructField> fields = ((StructType) boundSlot.getDataType()).getFields();
+ for (int idx = 0; idx < fields.size(); ++idx) {
+ expandAlias.add(new Alias(new StructElement(
+ boundSlot, new StringLiteral(fields.get(idx).getName())),
+ generate.getExpandColumnAlias().get(i).get(idx)));
+ }
+ }
+ }
+ LogicalGenerate ret = new LogicalGenerate<>(
+ boundFunctionGenerators, slotBuilder.build(), generate.child());
+ if (expandAlias.size() > 0) {
+ // we need a project to deal with explode(map) to struct with field alias
+ // project should contains: generator.child slot + expandAlias
+ List<NamedExpression> allProjectSlots = generate.child().getOutput().stream()
+ .map(NamedExpression.class::cast)
+ .collect(Collectors.toList());
+ allProjectSlots.addAll(expandAlias);
+ return new LogicalProject<>(allProjectSlots, ret);
Review Comment:
```suggestion
ret = new LogicalProject<>(allProjectSlots, ret);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java:
##########
@@ -587,7 +593,7 @@ protected boolean condition(Rule rule, Plan plan) {
// we need to do cast before set operation, because we maybe use these slot to do shuffle
// so, we must cast it before shuffle to get correct hash code.
List<List<NamedExpression>> childrenProjections = setOperation.collectChildrenProjections();
- ImmutableList.Builder<List<SlotReference>> childrenOutputs = ImmutableList.builder();
+ Builder<List<SlotReference>> childrenOutputs = ImmutableList.builder();
Review Comment:
do not refactor other code in this PR
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -1021,14 +1021,21 @@ protected LogicalPlan withGenerate(LogicalPlan plan, LateralViewContext ctx) {
return plan;
}
String generateName = ctx.tableName.getText();
- String columnName = ctx.columnName.getText();
+ // if later view explode map type, we need to add a project to convert map to struct
+ String columnName = ctx.columnNames.get(0).getText();
+ List<String> expandColumnNames = Lists.newArrayList();
+ if (ctx.columnNames.size() > 1) {
+ columnName = "expand_cols";
Review Comment:
use column name generator to generate column name when connect context not null.
```java
ConnectContext.get().getStatementContext().generateColumnName()
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/generator/ExplodeMapOuter.java:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.generator;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.MapType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * explode({"amory":1, "doris": 2}) generate two column and two lines with:
+ * key column: amory, doris
+ * value column: 1, 2
+ */
+public class ExplodeMapOuter extends TableGeneratingFunction implements UnaryExpression, AlwaysNullable {
+
+ /**
+ * constructor with 1 argument.
+ */
+ public ExplodeMapOuter(Expression arg) {
+ super("explode_map_outer", arg);
+ }
+
+ /**
+ * withChildren.
+ */
+ @Override
+ public ExplodeMapOuter withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new ExplodeMapOuter(children.get(0));
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (!(child().getDataType() instanceof MapType)) {
+ throw new AnalysisException("only support map type for explode_map function but got "
+ + child().getDataType());
+ }
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return ImmutableList.of(
+ FunctionSignature.ret(new StructType(ImmutableList.of(
+ new StructField("col1", ((MapType) child().getDataType()).getKeyType(), true, ""),
Review Comment:
create a function in structType like: `StructType constructStructType(List<DataType> fieldTypes)` to generate default column name struct type. and call this function here and in `org.apache.doris.nereids.trees.expressions.literal.StructLiteral#computeDataType`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java:
##########
@@ -624,8 +631,34 @@ protected boolean condition(Rule rule, Plan plan) {
Slot boundSlot = new SlotReference(slot.getNameParts().get(1), generator.getDataType(),
generator.nullable(), ImmutableList.of(slot.getNameParts().get(0)));
slotBuilder.add(boundSlot);
+ // the boundSlot may has two situation:
+ // 1. the expandColumnsAlias is not empty, we should use make boundSlot expand to multi alias
+ // 2. the expandColumnsAlias is empty, we should use origin boundSlot
+ if (generate.getExpandColumnAlias() != null
+ && !CollectionUtils.isEmpty(generate.getExpandColumnAlias().get(i))) {
Review Comment:
`generate.getExpandColumnAlias().get(i)` could generate OutOfBound exception, should check first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958601653
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 59d4ae2d3358788a223fce971e8cc2983ad56620 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1953642044
<details>
<summary>TPC-DS: <b>Total hot run time: 176671 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d09e6481faba217bcd588c792b6ffd773eeeefb, data reload: false
query1 913 342 341 341
query2 6550 1776 1771 1771
query3 6696 203 198 198
query4 23052 21095 21097 21095
query5 4204 378 369 369
query6 255 164 162 162
query7 4605 294 293 293
query8 241 191 205 191
query9 8423 2801 2771 2771
query10 414 216 216 216
query11 15026 14403 14396 14396
query12 130 83 81 81
query13 1680 415 412 412
query14 9031 7517 7482 7482
query15 216 190 182 182
query16 7364 258 255 255
query17 1398 570 533 533
query18 1969 269 257 257
query19 193 143 144 143
query20 88 82 84 82
query21 190 114 115 114
query22 5108 4809 4801 4801
query23 32461 31468 31491 31468
query24 11812 3437 3341 3341
query25 610 362 362 362
query26 1523 150 161 150
query27 3065 307 313 307
query28 6600 1841 1825 1825
query29 1110 597 600 597
query30 271 136 152 136
query31 932 731 751 731
query32 89 54 54 54
query33 712 242 237 237
query34 1045 483 485 483
query35 923 821 821 821
query36 985 900 887 887
query37 134 62 60 60
query38 3299 3181 3211 3181
query39 1357 1327 1305 1305
query40 281 103 105 103
query41 38 35 35 35
query42 102 97 96 96
query43 483 454 457 454
query44 1085 685 701 685
query45 195 180 175 175
query46 1048 766 795 766
query47 1626 1527 1573 1527
query48 421 338 347 338
query49 1110 298 292 292
query50 774 364 367 364
query51 5321 5166 5256 5166
query52 106 98 89 89
query53 392 293 295 293
query54 291 222 233 222
query55 82 80 76 76
query56 216 197 200 197
query57 1070 944 952 944
query58 217 199 199 199
query59 2325 2219 2181 2181
query60 246 212 211 211
query61 82 82 88 82
query62 583 390 366 366
query63 321 277 280 277
query64 6461 3067 3083 3067
query65 3289 3249 3218 3218
query66 1331 324 319 319
query67 14739 14404 14021 14021
query68 5088 559 532 532
query69 508 355 347 347
query70 1252 1257 1196 1196
query71 340 251 246 246
query72 6323 2750 2607 2607
query73 692 307 307 307
query74 6949 6437 6491 6437
query75 3167 2561 2514 2514
query76 3207 1128 1200 1128
query77 343 232 230 230
query78 9423 8881 8725 8725
query79 1014 512 491 491
query80 694 347 330 330
query81 439 203 208 203
query82 1198 83 82 82
query83 231 120 122 120
query84 223 78 76 76
query85 1004 336 331 331
query86 320 309 298 298
query87 3421 3243 3287 3243
query88 2681 2270 2256 2256
query89 419 349 340 340
query90 2200 168 163 163
query91 154 128 125 125
query92 52 48 46 46
query93 1029 488 508 488
query94 1239 180 181 180
query95 8632 371 8313 371
query96 592 261 268 261
query97 4436 4243 4284 4243
query98 209 203 194 194
query99 1115 748 708 708
Total cold run time: 277503 ms
Total hot run time: 176671 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495503862
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/generator/ExplodeMapOuter.java:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.generator;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.MapType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * explode({"amory":1, "doris": 2}) generate two column and two lines with:
+ * key column: amory, doris
+ * value column: 1, 2
+ */
+public class ExplodeMapOuter extends TableGeneratingFunction implements UnaryExpression, AlwaysNullable {
+
+ /**
+ * constructor with 1 argument.
+ */
+ public ExplodeMapOuter(Expression arg) {
+ super("explode_map_outer", arg);
+ }
+
+ /**
+ * withChildren.
+ */
+ @Override
+ public ExplodeMapOuter withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new ExplodeMapOuter(children.get(0));
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (!(child().getDataType() instanceof MapType)) {
+ throw new AnalysisException("only support map type for explode_map function but got "
+ + child().getDataType());
+ }
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return ImmutableList.of(
+ FunctionSignature.ret(new StructType(ImmutableList.of(
+ new StructField("col1", ((MapType) child().getDataType()).getKeyType(), true, ""),
Review Comment:
yes, this is struct default col name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495571851
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java:
##########
@@ -1929,6 +1930,15 @@ private void initTableFunction() {
addTableFunctionWithCombinator(EXPLODE, Type.WILDCARD_DECIMAL, Function.NullableMode.ALWAYS_NULLABLE,
Lists.newArrayList(new ArrayType(Type.WILDCARD_DECIMAL)), false,
"_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
+
+ initTableFunctionListWithCombinator(EXPLODE_MAP);
Review Comment:
it's usefull
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958595702
<details>
<summary>TPC-DS: <b>Total hot run time: 177323 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59d4ae2d3358788a223fce971e8cc2983ad56620, data reload: false
query1 915 345 339 339
query2 6520 1735 1840 1735
query3 6700 206 209 206
query4 23129 21240 21234 21234
query5 4263 386 375 375
query6 259 175 171 171
query7 4608 297 295 295
query8 252 201 203 201
query9 8405 2879 2860 2860
query10 423 216 236 216
query11 15020 14668 14497 14497
query12 140 83 81 81
query13 1682 413 426 413
query14 9105 7732 7776 7732
query15 208 186 191 186
query16 7296 253 250 250
query17 1415 546 525 525
query18 1972 276 258 258
query19 198 148 150 148
query20 87 84 85 84
query21 184 129 114 114
query22 4941 4743 4773 4743
query23 32614 31572 31576 31572
query24 12720 3428 3459 3428
query25 614 364 368 364
query26 1787 166 159 159
query27 3042 312 320 312
query28 6804 1865 1850 1850
query29 1172 622 613 613
query30 272 140 144 140
query31 931 741 757 741
query32 100 65 60 60
query33 722 233 230 230
query34 1070 487 501 487
query35 924 816 828 816
query36 1017 856 893 856
query37 140 63 59 59
query38 3264 3226 3220 3220
query39 1408 1331 1331 1331
query40 282 103 107 103
query41 37 34 34 34
query42 106 101 100 100
query43 476 442 455 442
query44 1066 683 699 683
query45 202 186 177 177
query46 1059 813 792 792
query47 1657 1539 1590 1539
query48 413 346 350 346
query49 1158 310 308 308
query50 778 379 374 374
query51 4491 4343 4344 4343
query52 110 96 105 96
query53 393 303 295 295
query54 301 230 224 224
query55 88 83 83 83
query56 230 208 202 202
query57 1064 945 940 940
query58 219 202 199 199
query59 2325 2226 2257 2226
query60 246 230 217 217
query61 85 86 81 81
query62 590 374 381 374
query63 329 291 283 283
query64 6465 3069 3129 3069
query65 3297 3251 3270 3251
query66 1364 340 318 318
query67 14475 14332 14287 14287
query68 5100 552 576 552
query69 527 357 356 356
query70 1264 1192 1203 1192
query71 380 257 252 252
query72 6344 2762 2608 2608
query73 712 307 310 307
query74 6878 6396 6410 6396
query75 3198 2576 2562 2562
query76 3216 1149 1218 1149
query77 365 239 234 234
query78 9560 8871 8779 8779
query79 972 510 504 504
query80 517 354 341 341
query81 448 201 207 201
query82 160 87 87 87
query83 144 128 119 119
query84 223 75 80 75
query85 1040 355 342 342
query86 296 314 296 296
query87 3416 3311 3251 3251
query88 2678 2279 2277 2277
query89 445 365 369 365
query90 1968 165 165 165
query91 149 129 131 129
query92 54 55 48 48
query93 1039 527 498 498
query94 1134 178 180 178
query95 442 335 343 335
query96 577 268 266 266
query97 4407 4290 4285 4285
query98 222 199 198 198
query99 1081 689 718 689
Total cold run time: 268585 ms
Total hot run time: 177323 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958994882
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1953648803
<details>
<summary>ClickBench: <b>Total hot run time: 30.55 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d09e6481faba217bcd588c792b6ffd773eeeefb, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.03 0.03
query3 0.23 0.07 0.08
query4 1.66 0.08 0.07
query5 0.49 0.47 0.48
query6 1.36 0.64 0.61
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.53 0.46 0.43
query10 0.49 0.49 0.49
query11 0.12 0.09 0.10
query12 0.11 0.10 0.10
query13 0.58 0.59 0.58
query14 0.75 0.79 0.77
query15 0.81 0.79 0.80
query16 0.33 0.36 0.34
query17 0.90 0.90 0.89
query18 0.17 0.15 0.20
query19 1.81 1.64 1.68
query20 0.01 0.00 0.01
query21 15.40 0.67 0.60
query22 2.99 4.26 2.15
query23 17.52 1.03 0.97
query24 2.01 0.75 0.36
query25 0.62 0.06 0.05
query26 0.16 0.15 0.13
query27 0.07 0.04 0.06
query28 11.83 0.80 0.83
query29 12.66 3.31 3.28
query30 0.51 0.48 0.49
query31 2.78 0.34 0.38
query32 3.34 0.46 0.47
query33 3.18 3.15 3.13
query34 15.36 4.48 4.50
query35 4.49 4.48 4.50
query36 1.07 0.94 0.96
query37 0.08 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.17 0.14 0.14
query41 0.07 0.02 0.02
query42 0.03 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 104.92 s
Total hot run time: 30.55 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1953649009
TeamCity be ut coverage result:
Function Coverage: 35.75% (8558/23937)
Line Coverage: 27.71% (69419/250510)
Region Coverage: 26.83% (36020/134265)
Branch Coverage: 23.65% (18427/77928)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d09e6481faba217bcd588c792b6ffd773eeeefb_2d09e6481faba217bcd588c792b6ffd773eeeefb/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1955811283
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1955815235
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1956063025
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958599419
<details>
<summary>ClickBench: <b>Total hot run time: 31.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59d4ae2d3358788a223fce971e8cc2983ad56620, data reload: false
query1 0.03 0.03 0.02
query2 0.06 0.02 0.02
query3 0.22 0.07 0.07
query4 1.66 0.09 0.11
query5 0.50 0.48 0.50
query6 1.38 0.62 0.61
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.54 0.45 0.45
query10 0.49 0.50 0.49
query11 0.14 0.10 0.09
query12 0.13 0.10 0.10
query13 0.59 0.59 0.60
query14 0.77 0.81 0.78
query15 0.82 0.80 0.80
query16 0.34 0.34 0.34
query17 0.92 0.92 0.88
query18 0.18 0.16 0.15
query19 1.86 1.65 1.71
query20 0.02 0.01 0.02
query21 15.40 0.65 0.57
query22 2.80 4.28 3.09
query23 17.52 1.07 0.99
query24 2.08 0.87 0.12
query25 0.46 0.07 0.05
query26 0.17 0.13 0.14
query27 0.05 0.05 0.06
query28 12.03 0.81 0.82
query29 12.49 3.34 3.43
query30 0.52 0.48 0.49
query31 2.77 0.36 0.37
query32 3.36 0.48 0.49
query33 3.16 3.10 3.15
query34 15.36 4.51 4.51
query35 4.51 4.53 4.49
query36 1.09 0.97 0.96
query37 0.07 0.06 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.14 0.15
query41 0.06 0.01 0.02
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.87 s
Total hot run time: 31.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1953652687
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 2d09e6481faba217bcd588c792b6ffd773eeeefb with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1916495564
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1916571494
<details>
<summary>TPC-DS: <b>Total hot run time: 172186 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1be87bdad54ff870d146fa9f596f94b21732d696, data reload: false
query1 930 331 322 322
query2 6575 1953 1894 1894
query3 6697 204 202 202
query4 27475 22013 21975 21975
query5 4461 348 413 348
query6 254 160 153 153
query7 4610 259 252 252
query8 250 175 177 175
query9 9096 2239 2234 2234
query10 422 200 196 196
query11 17023 15308 15213 15213
query12 124 72 63 63
query13 1674 368 360 360
query14 9246 6464 6466 6464
query15 220 175 178 175
query16 6349 256 252 252
query17 1884 481 462 462
query18 1911 255 246 246
query19 177 131 141 131
query20 66 71 67 67
query21 194 124 129 124
query22 5082 4850 4610 4610
query23 31028 30269 30190 30190
query24 11048 2836 2803 2803
query25 542 310 300 300
query26 835 137 139 137
query27 2785 279 280 279
query28 6139 1861 1816 1816
query29 925 620 614 614
query30 279 136 138 136
query31 931 695 719 695
query32 88 47 47 47
query33 595 202 207 202
query34 835 451 454 451
query35 848 742 732 732
query36 1363 1248 1149 1149
query37 89 55 58 55
query38 3260 3167 3149 3149
query39 1298 1237 1248 1237
query40 194 83 79 79
query41 36 36 33 33
query42 82 81 86 81
query43 515 488 484 484
query44 1058 673 686 673
query45 188 177 174 174
query46 1030 631 652 631
query47 1608 1546 1508 1508
query48 369 317 301 301
query49 1120 279 276 276
query50 693 304 309 304
query51 5234 5118 5140 5118
query52 92 75 79 75
query53 330 253 257 253
query54 268 179 190 179
query55 73 75 70 70
query56 172 160 155 155
query57 973 932 891 891
query58 182 149 160 149
query59 2428 2305 2503 2305
query60 202 185 171 171
query61 83 85 82 82
query62 574 369 376 369
query63 283 255 253 253
query64 4146 1740 1699 1699
query65 3284 3226 3199 3199
query66 1127 324 308 308
query67 14413 14349 14138 14138
query68 4134 495 486 486
query69 438 308 287 287
query70 1615 1583 1594 1583
query71 280 205 209 205
query72 3796 2817 2841 2817
query73 679 321 310 310
query74 6555 6235 6272 6235
query75 2809 2264 2245 2245
query76 2979 957 934 934
query77 367 229 230 229
query78 9058 8794 8518 8518
query79 3022 507 484 484
query80 1940 309 324 309
query81 543 189 198 189
query82 677 80 78 78
query83 350 115 118 115
query84 280 73 74 73
query85 1752 353 320 320
query86 508 368 377 368
query87 3468 3322 3241 3241
query88 3713 2253 2298 2253
query89 423 365 360 360
query90 1950 187 183 183
query91 147 117 116 116
query92 54 42 42 42
query93 2366 440 424 424
query94 1388 163 155 155
query95 491 446 454 446
query96 628 332 316 316
query97 4242 4124 4138 4124
query98 203 206 184 184
query99 1043 704 689 689
Total cold run time: 270263 ms
Total hot run time: 172186 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1470677473
##########
be/src/vec/exprs/table_function/vexplode_map.cpp:
##########
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "vec/exprs/table_function/vexplode_map.h"
+
+#include <glog/logging.h>
+
+#include <ostream>
+#include <vector>
+
+#include "common/status.h"
+#include "vec/columns/column.h"
+#include "vec/common/string_ref.h"
+#include "vec/core/block.h"
+#include "vec/core/column_with_type_and_name.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vexpr_context.h"
+
+namespace doris::vectorized {
+
+VExplodeMapTableFunction::VExplodeMapTableFunction() {
Review Comment:
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
```cpp
VExplodeMapTableFunction::VExplodeMapTableFunction() {
^
```
##########
be/src/vec/exprs/table_function/vexplode_map.h:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
Review Comment:
warning: inclusion of deprecated C++ header 'stddef.h'; consider using 'cstddef' instead [modernize-deprecated-headers]
```suggestion
#include <cstddef>
```
##########
be/src/vec/exprs/table_function/vexplode_map.h:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+
+#include "common/status.h"
+#include "vec/columns/column_map.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_map.h"
+#include "vec/exprs/table_function/table_function.h"
+#include "vec/functions/array/function_array_utils.h"
+
+namespace doris {
+namespace vectorized {
Review Comment:
warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]
```suggestion
namespace doris::vectorized {
```
be/src/vec/exprs/table_function/vexplode_map.h:32:
```diff
- } // namespace vectorized
- } // namespace doris
+ } // namespace doris
```
##########
be/src/vec/exprs/table_function/vexplode_map.h:
##########
@@ -0,0 +1,72 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stddef.h>
+
+#include "common/status.h"
+#include "vec/columns/column_map.h"
+#include "vec/data_types/data_type.h"
+#include "vec/data_types/data_type_array.h"
+#include "vec/data_types/data_type_map.h"
+#include "vec/exprs/table_function/table_function.h"
+#include "vec/functions/array/function_array_utils.h"
+
+namespace doris {
+namespace vectorized {
+class Block;
+} // namespace vectorized
+} // namespace doris
+
+namespace doris::vectorized {
+
+class ColumnMapExecutionData {
+public:
+ void reset() {
+ map_nullmap_data = nullptr;
+ map_col = nullptr;
+ offsets_ptr = nullptr;
+ }
+
+public:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/vec/exprs/table_function/vexplode_map.h:38:** previously declared here
```cpp
public:
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1916582120
<details>
<summary>ClickBench: <b>Total hot run time: 31.28 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1be87bdad54ff870d146fa9f596f94b21732d696, data reload: false
query1 0.04 0.04 0.03
query2 0.05 0.02 0.02
query3 0.22 0.06 0.06
query4 1.68 0.09 0.10
query5 0.52 0.51 0.51
query6 1.18 0.65 0.63
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.57 0.50 0.48
query10 0.57 0.55 0.56
query11 0.12 0.09 0.08
query12 0.10 0.09 0.10
query13 0.60 0.61 0.60
query14 0.79 0.80 0.78
query15 0.79 0.78 0.76
query16 0.38 0.38 0.37
query17 1.03 0.98 1.01
query18 0.20 0.25 0.24
query19 1.82 1.74 1.79
query20 0.01 0.01 0.01
query21 15.40 0.59 0.56
query22 2.79 2.42 1.97
query23 17.57 0.84 0.90
query24 2.24 1.22 1.37
query25 0.36 0.23 0.15
query26 0.53 0.14 0.14
query27 0.04 0.04 0.04
query28 10.61 0.86 0.84
query29 12.60 3.23 3.16
query30 0.65 0.54 0.48
query31 2.78 0.34 0.35
query32 3.38 0.48 0.48
query33 3.20 3.18 3.27
query34 15.95 4.25 4.27
query35 4.28 4.27 4.27
query36 1.10 1.04 1.06
query37 0.06 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.01 0.02
query40 0.15 0.13 0.13
query41 0.07 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.58 s
Total hot run time: 31.28 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1497019624
##########
regression-test/suites/nereids_p0/sql_functions/table_function/explode_map.groovy:
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("explode_map") {
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+
+ sql """ DROP TABLE IF EXISTS sdu """
+ sql """
+ CREATE TABLE IF NOT EXISTS `sdu`(
+ `id` INT NULL,
+ `name` TEXT NULL,
+ `score` MAP<TEXT,INT> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ // insert values
+ sql """ insert into sdu values (0, "zhangsan", {"Chinese":"80","Math":"60","English":"90"}); """
+ sql """ insert into sdu values (1, "lisi", {"null":null}); """
+ sql """ insert into sdu values (2, "wangwu", {"Chinese":"88","Math":"90","English":"96"}); """
+ sql """ insert into sdu values (3, "lisi2", {null:null}); """
+ sql """ insert into sdu values (4, "amory", NULL); """
+
+ qt_sql """ select * from sdu order by id; """
+ qt_explode_sql """ select name, k,v from sdu lateral view explode_map(score) tmp as k,v order by id;"""
Review Comment:
id is not in select list， why not throw exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1497035651
##########
regression-test/suites/nereids_p0/sql_functions/table_function/explode_map.groovy:
##########
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("explode_map") {
+ sql "SET enable_nereids_planner=true"
+ sql "SET enable_fallback_to_original_planner=false"
+
+ sql """ DROP TABLE IF EXISTS sdu """
+ sql """
+ CREATE TABLE IF NOT EXISTS `sdu`(
+ `id` INT NULL,
+ `name` TEXT NULL,
+ `score` MAP<TEXT,INT> NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1");
+ """
+
+ // insert values
+ sql """ insert into sdu values (0, "zhangsan", {"Chinese":"80","Math":"60","English":"90"}); """
+ sql """ insert into sdu values (1, "lisi", {"null":null}); """
+ sql """ insert into sdu values (2, "wangwu", {"Chinese":"88","Math":"90","English":"96"}); """
+ sql """ insert into sdu values (3, "lisi2", {null:null}); """
+ sql """ insert into sdu values (4, "amory", NULL); """
+
+ qt_sql """ select * from sdu order by id; """
+ qt_explode_sql """ select name, k,v from sdu lateral view explode_map(score) tmp as k,v order by id;"""
Review Comment:
lateral view explode is table function, which has all column info , and this is same action with explode too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1916577292
TeamCity be ut coverage result:
Function Coverage: 36.18% (8626/23845)
Line Coverage: 28.24% (70532/249771)
Region Coverage: 27.23% (36391/133620)
Branch Coverage: 24.02% (18636/77582)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1be87bdad54ff870d146fa9f596f94b21732d696_1be87bdad54ff870d146fa9f596f94b21732d696/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958571787
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30151:
URL: https://github.com/apache/doris/pull/30151#issuecomment-1958588136
<details>
<summary>TPC-H: <b>Total hot run time: 41701 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59d4ae2d3358788a223fce971e8cc2983ad56620, data reload: false
------ Round 1 ----------------------------------
q1 19249 5255 5133 5133
q2 3016 143 141 141
q3 11555 1034 1031 1031
q4 4978 969 981 969
q5 7635 3172 3224 3172
q6 199 136 133 133
q7 1287 777 775 775
q8 9244 2105 2099 2099
q9 7576 6649 6610 6610
q10 8314 2635 2633 2633
q11 421 219 221 219
q12 832 326 333 326
q13 17936 3624 3644 3624
q14 324 261 258 258
q15 587 514 501 501
q16 480 410 406 406
q17 931 890 877 877
q18 7248 6698 6717 6698
q19 1568 1510 1501 1501
q20 476 267 264 264
q21 6165 4039 3996 3996
q22 553 335 344 335
Total cold run time: 110574 ms
Total hot run time: 41701 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4888 4847 4867 4847
q2 295 188 184 184
q3 3622 3616 3614 3614
q4 2568 2562 2550 2550
q5 5790 5767 5779 5767
q6 213 128 124 124
q7 2272 1646 1679 1646
q8 3085 3109 3144 3109
q9 8767 8734 8756 8734
q10 4230 4254 4262 4254
q11 526 376 370 370
q12 762 545 548 545
q13 4172 3428 3429 3428
q14 264 238 255 238
q15 597 509 545 509
q16 481 439 435 435
q17 1690 1640 1628 1628
q18 8318 7694 7684 7684
q19 1666 1669 1655 1655
q20 2120 1850 1854 1850
q21 7065 6236 6205 6205
q22 577 514 507 507
Total cold run time: 63968 ms
Total hot run time: 59883 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495494561
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/generator/ExplodeMapOuter.java:
##########
@@ -0,0 +1,80 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.generator;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNullable;
+import org.apache.doris.nereids.trees.expressions.shape.UnaryExpression;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.MapType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * explode({"amory":1, "doris": 2}) generate two column and two lines with:
+ * key column: amory, doris
+ * value column: 1, 2
+ */
+public class ExplodeMapOuter extends TableGeneratingFunction implements UnaryExpression, AlwaysNullable {
+
+ /**
+ * constructor with 1 argument.
+ */
+ public ExplodeMapOuter(Expression arg) {
+ super("explode_map_outer", arg);
+ }
+
+ /**
+ * withChildren.
+ */
+ @Override
+ public ExplodeMapOuter withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new ExplodeMapOuter(children.get(0));
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (!(child().getDataType() instanceof MapType)) {
+ throw new AnalysisException("only support map type for explode_map function but got "
+ + child().getDataType());
+ }
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return ImmutableList.of(
+ FunctionSignature.ret(new StructType(ImmutableList.of(
+ new StructField("col1", ((MapType) child().getDataType()).getKeyType(), true, ""),
Review Comment:
Does "col1"/"col2" is properl?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](explode) support explode map type [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #30151:
URL: https://github.com/apache/doris/pull/30151#discussion_r1495498956
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/FunctionSet.java:
##########
@@ -1929,6 +1930,15 @@ private void initTableFunction() {
addTableFunctionWithCombinator(EXPLODE, Type.WILDCARD_DECIMAL, Function.NullableMode.ALWAYS_NULLABLE,
Lists.newArrayList(new ArrayType(Type.WILDCARD_DECIMAL)), false,
"_ZN5doris19DummyTableFunctions7explodeEPN9doris_udf15FunctionContextERKNS1_13CollectionValE");
+
+ initTableFunctionListWithCombinator(EXPLODE_MAP);
Review Comment:
Are those code useless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org