You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zclllyybb (via GitHub)" <gi...@apache.org> on 2024/03/05 18:27:26 UTC
[PR] [Feature-WIP](partition) Suppor null range partition [doris]
zclllyybb opened a new pull request, #31827:
URL: https://github.com/apache/doris/pull/31827
## Proposed changes
Issue Number: close #xxx
still some bug to fix
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1981224615
TeamCity be ut coverage result:
Function Coverage: 35.41% (8559/24174)
Line Coverage: 27.16% (69485/255824)
Region Coverage: 26.37% (36034/136643)
Branch Coverage: 23.21% (18419/79344)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d426c543e1be2c8ba3a8a5d4b01b350638ea36b7_d426c543e1be2c8ba3a8a5d4b01b350638ea36b7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014043830
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1529671014
##########
be/src/vec/columns/column_nullable.cpp:
##########
@@ -390,17 +391,23 @@ int ColumnNullable::compare_at(size_t n, size_t m, const IColumn& rhs_,
/// instead of the usual nan_direction_hint and is used to implement
/// the ordering specified by either NULLS FIRST or NULLS LAST in the
/// ORDER BY construction.
- const auto& nullable_rhs = assert_cast<const ColumnNullable&>(rhs_);
Review Comment:
The way change the default behavior in Doris， `compare_at` must use same type column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1981023114
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1514005638
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -247,29 +256,34 @@ Status VRowDistribution::_generate_rows_distribution_for_auto_partition(
if (!_missing_map.empty()) {
// for missing partition keys, calc the missing partition and save in _partitions_need_create
- auto [part_ctxs, part_funcs] = _get_partition_function();
- auto funcs_size = part_funcs.size();
+ auto [part_ctxs, part_exprs] = _get_partition_function();
+ auto part_col_num = part_exprs.size();
+ // the two vectors are in column-first-order
std::vector<std::vector<std::string>> col_strs;
- col_strs.resize(funcs_size);
-
- for (int i = 0; i < funcs_size; ++i) {
- auto return_type = part_funcs[i]->data_type();
- // expose the data column
- vectorized::ColumnPtr range_left_col =
- block->get_by_position(partition_cols_idx[i]).column;
- if (const auto* nullable =
- check_and_get_column<vectorized::ColumnNullable>(*range_left_col)) {
- range_left_col = nullable->get_nested_column_ptr();
- return_type = assert_cast<const vectorized::DataTypeNullable*>(return_type.get())
- ->get_nested_type();
+ std::vector<const NullMap*> col_null_maps;
+ col_strs.resize(part_col_num);
+ col_null_maps.reserve(part_col_num);
Review Comment:
why not direct call reize ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014341139
<details>
<summary>TPC-H: <b>Total hot run time: 38044 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8b90b2d9f94602c21555f976383e2633d778c56, data reload: false
------ Round 1 ----------------------------------
q1 17619 4361 4139 4139
q2 2031 158 147 147
q3 10667 1150 1212 1150
q4 10642 766 780 766
q5 7466 3032 2956 2956
q6 201 123 121 121
q7 1059 596 577 577
q8 9521 2054 2018 2018
q9 7254 6602 6489 6489
q10 8431 3437 3545 3437
q11 431 222 218 218
q12 370 193 198 193
q13 17817 2848 2879 2848
q14 228 202 206 202
q15 507 470 458 458
q16 494 365 363 363
q17 985 645 536 536
q18 7278 6520 6606 6520
q19 2387 1444 1453 1444
q20 538 257 256 256
q21 3568 2904 3059 2904
q22 353 302 307 302
Total cold run time: 109847 ms
Total hot run time: 38044 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4215 4098 4094 4094
q2 330 231 230 230
q3 2963 2874 2846 2846
q4 1878 1563 1541 1541
q5 5286 5293 5240 5240
q6 196 116 117 116
q7 2259 1851 1834 1834
q8 3188 3334 3320 3320
q9 8618 8603 8609 8603
q10 3688 3726 3701 3701
q11 566 444 450 444
q12 727 543 565 543
q13 16897 2878 2826 2826
q14 287 266 249 249
q15 494 446 458 446
q16 484 418 403 403
q17 1758 1507 1456 1456
q18 7645 7234 7185 7185
q19 1657 1566 1542 1542
q20 1925 1734 1707 1707
q21 4825 4646 4639 4639
q22 545 456 441 441
Total cold run time: 70431 ms
Total hot run time: 53406 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014122257
<details>
<summary>TPC-H: <b>Total hot run time: 37543 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f4a4904f509eb83193efb1bf29d516f9062ed9a, data reload: false
------ Round 1 ----------------------------------
q1 17655 4359 4145 4145
q2 2020 155 147 147
q3 10588 1148 1188 1148
q4 10222 814 766 766
q5 7461 3014 2906 2906
q6 207 124 120 120
q7 1047 574 560 560
q8 9347 1980 1994 1980
q9 7142 6514 6439 6439
q10 8398 3331 3491 3331
q11 435 219 211 211
q12 412 193 194 193
q13 17798 2854 2828 2828
q14 225 197 205 197
q15 498 454 453 453
q16 464 357 363 357
q17 968 584 567 567
q18 7265 6450 6476 6450
q19 1992 1435 1491 1435
q20 554 261 242 242
q21 3523 2888 2777 2777
q22 368 291 305 291
Total cold run time: 108589 ms
Total hot run time: 37543 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4109 4074 4067 4067
q2 322 226 227 226
q3 2935 2777 2857 2777
q4 1865 1553 1563 1553
q5 5235 5270 5261 5261
q6 190 114 115 114
q7 2227 1852 1794 1794
q8 3138 3301 3262 3262
q9 8601 8583 8549 8549
q10 3692 3690 3709 3690
q11 535 437 428 428
q12 723 515 584 515
q13 16925 2866 2896 2866
q14 282 248 260 248
q15 482 446 444 444
q16 445 408 415 408
q17 1728 1507 1470 1470
q18 7464 7141 7102 7102
q19 1621 1583 1514 1514
q20 1918 1679 1726 1679
q21 4803 4750 4574 4574
q22 520 442 423 423
Total cold run time: 69760 ms
Total hot run time: 52964 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1984269695
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004357359
TeamCity be ut coverage result:
Function Coverage: 35.25% (8701/24687)
Line Coverage: 27.08% (71195/262924)
Region Coverage: 26.33% (36928/140276)
Branch Coverage: 23.24% (18886/81272)
Coverage Report: http://coverage.selectdb-in.cc/coverage/187c164f59c8d94bdf319a5f71aab22cae9dd698_187c164f59c8d94bdf319a5f71aab22cae9dd698/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2008589631
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1534851392
##########
be/src/exec/tablet_info.cpp:
##########
@@ -551,16 +504,84 @@ Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr,
part_key->second = column->size() - 1;
return Status::OK();
}
+// NOLINTEND(readability-function-size)
+
+Status VOlapTablePartitionParam::_create_partition_keys(const std::vector<TExprNode>& t_exprs,
Review Comment:
warning: method '_create_partition_keys' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status VOlapTablePartitionParam::_create_partition_keys(const std::vector<TExprNode>& t_exprs,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014282173
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1984940862
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2015475047
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014349651
<details>
<summary>TPC-DS: <b>Total hot run time: 181189 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8b90b2d9f94602c21555f976383e2633d778c56, data reload: false
query1 945 369 350 350
query2 7459 2082 1925 1925
query3 6709 209 210 209
query4 31189 20756 20606 20606
query5 4333 405 395 395
query6 258 177 170 170
query7 4621 297 285 285
query8 235 163 165 163
query9 9392 2305 2299 2299
query10 466 243 271 243
query11 15349 14091 14526 14091
query12 142 84 89 84
query13 1622 412 410 410
query14 11917 10919 10488 10488
query15 262 193 187 187
query16 8157 264 257 257
query17 1974 577 549 549
query18 1889 294 287 287
query19 342 154 162 154
query20 92 89 91 89
query21 197 128 130 128
query22 4856 4594 4580 4580
query23 31847 31087 31342 31087
query24 10675 2752 2829 2752
query25 610 384 393 384
query26 1128 157 159 157
query27 2960 357 359 357
query28 7554 1915 1902 1902
query29 883 650 631 631
query30 302 151 151 151
query31 967 731 733 731
query32 93 58 57 57
query33 762 246 334 246
query34 972 487 478 478
query35 831 611 638 611
query36 990 857 858 857
query37 128 78 80 78
query38 3558 3428 3444 3428
query39 1408 1384 1383 1383
query40 207 120 117 117
query41 51 48 47 47
query42 105 98 96 96
query43 473 441 451 441
query44 1165 724 711 711
query45 273 263 269 263
query46 1098 681 673 673
query47 1678 1604 1621 1604
query48 454 385 357 357
query49 1102 336 351 336
query50 776 371 365 365
query51 6606 6569 6650 6569
query52 102 94 90 90
query53 347 278 273 273
query54 314 237 266 237
query55 85 83 83 83
query56 254 231 230 230
query57 1103 1016 1022 1016
query58 241 211 212 211
query59 2668 2561 2539 2539
query60 286 254 263 254
query61 115 133 137 133
query62 602 396 419 396
query63 301 275 282 275
query64 5828 4064 3861 3861
query65 3068 2995 2982 2982
query66 1381 350 337 337
query67 15165 14520 14372 14372
query68 7825 525 516 516
query69 625 379 385 379
query70 1239 1144 1120 1120
query71 519 280 275 275
query72 6480 2679 2513 2513
query73 739 317 327 317
query74 8254 6572 6600 6572
query75 4111 2803 2770 2770
query76 5052 931 945 931
query77 650 247 256 247
query78 10428 9920 9735 9735
query79 9861 512 520 512
query80 1511 385 372 372
query81 517 214 209 209
query82 696 199 194 194
query83 220 138 142 138
query84 289 80 78 78
query85 1296 323 315 315
query86 413 287 303 287
query87 3674 3543 3586 3543
query88 4442 2296 2297 2296
query89 493 363 361 361
query90 2055 172 175 172
query91 179 132 135 132
query92 60 49 48 48
query93 6855 500 488 488
query94 1184 177 172 172
query95 427 323 327 323
query96 611 272 264 264
query97 3067 2865 2876 2865
query98 226 209 203 203
query99 1121 752 735 735
Total cold run time: 309041 ms
Total hot run time: 181189 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1989295261
TeamCity be ut coverage result:
Function Coverage: 34.98% (8556/24459)
Line Coverage: 26.82% (69537/259275)
Region Coverage: 26.06% (36077/138438)
Branch Coverage: 23.03% (18442/80076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39eae89849e935e37c255c5415dfcd9077136d9d_39eae89849e935e37c255c5415dfcd9077136d9d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1979615789
TeamCity be ut coverage result:
Function Coverage: 35.42% (8556/24153)
Line Coverage: 27.17% (69440/255558)
Region Coverage: 26.39% (36025/136513)
Branch Coverage: 23.22% (18407/79262)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a48395ff99525094360aa2ce691ea99253c80199_a48395ff99525094360aa2ce691ea99253c80199/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1528707653
##########
be/src/vec/columns/column_const.h:
##########
@@ -21,11 +21,10 @@
#pragma once
#include <glog/logging.h>
Review Comment:
warning: 'glog/logging.h' file not found [clang-diagnostic-error]
```cpp
#include <glog/logging.h>
^
```
##########
be/src/vec/columns/column_const.h:
##########
@@ -199,22 +227,26 @@
size_t allocated_bytes() const override { return data->allocated_bytes() + sizeof(s); }
- int compare_at(size_t, size_t, const IColumn& rhs, int nan_direction_hint) const override {
- auto rhs_const_column = assert_cast<const ColumnConst&>(rhs);
+ int compare_at(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) const override {
Review Comment:
warning: method 'compare_at' can be made static [readability-convert-member-functions-to-static]
```suggestion
static int compare_at(size_t n, size_t m, const IColumn& rhs, int nan_direction_hint) override {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014137111
TeamCity be ut coverage result:
Function Coverage: 35.28% (8733/24753)
Line Coverage: 27.09% (71519/264037)
Region Coverage: 26.32% (37099/140932)
Branch Coverage: 23.22% (18973/81694)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8f4a4904f509eb83193efb1bf29d516f9062ed9a_8f4a4904f509eb83193efb1bf29d516f9062ed9a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014295975
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004319745
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1989275527
<details>
<summary>TPC-H: <b>Total hot run time: 38162 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d, data reload: false
------ Round 1 ----------------------------------
q1 17639 4107 4049 4049
q2 2026 150 145 145
q3 10554 943 929 929
q4 4667 937 948 937
q5 7577 2908 3023 2908
q6 181 126 126 126
q7 1281 826 823 823
q8 9492 2025 2080 2025
q9 7310 6531 6513 6513
q10 8221 2557 2549 2549
q11 420 218 213 213
q12 769 320 321 320
q13 17960 3003 2980 2980
q14 271 254 250 250
q15 504 466 446 446
q16 488 387 390 387
q17 965 904 880 880
q18 6710 6112 5885 5885
q19 1583 1532 1521 1521
q20 547 284 290 284
q21 7508 3691 3696 3691
q22 793 314 301 301
Total cold run time: 107466 ms
Total hot run time: 38162 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4007 4004 4009 4004
q2 319 221 233 221
q3 2958 2989 2926 2926
q4 1867 1837 1798 1798
q5 5242 5323 5239 5239
q6 202 117 124 117
q7 2266 1822 1814 1814
q8 3241 3300 3288 3288
q9 8523 8505 8496 8496
q10 6170 3715 3738 3715
q11 532 434 438 434
q12 693 516 556 516
q13 11831 2858 2816 2816
q14 275 251 252 251
q15 495 461 454 454
q16 433 403 395 395
q17 1705 1671 1676 1671
q18 7832 7641 7381 7381
q19 3140 1563 1613 1563
q20 1899 1680 1675 1675
q21 4936 4701 4824 4701
q22 501 446 463 446
Total cold run time: 69067 ms
Total hot run time: 53921 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004492069
TeamCity be ut coverage result:
Function Coverage: 35.25% (8701/24687)
Line Coverage: 27.08% (71194/262924)
Region Coverage: 26.32% (36922/140276)
Branch Coverage: 23.24% (18890/81272)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c2e1b523d35a6eb2874554ae384d080ace102793_c2e1b523d35a6eb2874554ae384d080ace102793/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1531461993
##########
be/src/vec/columns/column_const.h:
##########
@@ -218,6 +247,30 @@ class ColumnConst final : public COWHelper<IColumn, ColumnConst> {
}
}
+ int compare_concrete_at(size_t n, size_t m, const IColumn& rhs,
+ int nan_direction_hint) const override {
Review Comment:
warning: method 'compare_concrete_at' can be made static [readability-convert-member-functions-to-static]
```suggestion
static int compare_concrete_at(size_t n, size_t m, const IColumn& rhs,
int nan_direction_hint) override {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004105933
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1984465499
TeamCity be ut coverage result:
Function Coverage: 35.37% (8555/24186)
Line Coverage: 27.15% (69518/256052)
Region Coverage: 26.37% (36079/136843)
Branch Coverage: 23.21% (18442/79446)
Coverage Report: http://coverage.selectdb-in.cc/coverage/125e526d5bf857abc3558ad6d8f8c37fbc6dbbcd_125e526d5bf857abc3558ad6d8f8c37fbc6dbbcd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014368740
TeamCity be ut coverage result:
Function Coverage: 35.26% (8730/24756)
Line Coverage: 27.07% (71499/264151)
Region Coverage: 26.30% (37088/141003)
Branch Coverage: 23.21% (18970/81746)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8b90b2d9f94602c21555f976383e2633d778c56_a8b90b2d9f94602c21555f976383e2633d778c56/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1989314123
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 16.2 seconds inserted 10000000 Rows, about 617K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014131227
<details>
<summary>TPC-DS: <b>Total hot run time: 179765 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f4a4904f509eb83193efb1bf29d516f9062ed9a, data reload: false
query1 929 358 337 337
query2 7399 2029 1944 1944
query3 6701 223 217 217
query4 31224 20650 20792 20650
query5 4413 490 405 405
query6 261 176 170 170
query7 4620 289 284 284
query8 234 170 176 170
query9 9875 2201 2193 2193
query10 460 250 243 243
query11 17118 14065 14153 14065
query12 138 92 82 82
query13 1633 417 418 417
query14 11177 10166 10336 10166
query15 279 195 193 193
query16 8011 255 254 254
query17 1936 549 518 518
query18 2013 278 263 263
query19 208 160 146 146
query20 91 85 85 85
query21 192 130 122 122
query22 4630 4524 4500 4500
query23 31518 31030 30715 30715
query24 11214 2837 2820 2820
query25 632 378 380 378
query26 1270 149 160 149
query27 2947 347 337 337
query28 8021 1837 1860 1837
query29 896 641 627 627
query30 300 144 143 143
query31 961 710 720 710
query32 93 57 54 54
query33 763 252 248 248
query34 1100 487 491 487
query35 831 591 602 591
query36 961 860 873 860
query37 122 75 79 75
query38 3562 3487 3391 3391
query39 1424 1361 1359 1359
query40 208 111 112 111
query41 49 46 48 46
query42 104 95 95 95
query43 473 457 466 457
query44 1242 740 715 715
query45 268 259 256 256
query46 1089 676 687 676
query47 1704 1596 1599 1596
query48 432 358 354 354
query49 1139 335 350 335
query50 759 378 375 375
query51 6809 6656 6601 6601
query52 112 90 90 90
query53 338 270 270 270
query54 304 245 256 245
query55 80 81 82 81
query56 246 231 229 229
query57 1095 1018 1020 1018
query58 232 202 206 202
query59 2758 2632 2467 2467
query60 269 250 251 250
query61 117 113 121 113
query62 588 405 405 405
query63 296 268 269 268
query64 5406 3881 3904 3881
query65 3053 3003 2972 2972
query66 876 356 346 346
query67 15244 14350 14219 14219
query68 8600 531 529 529
query69 624 384 369 369
query70 1261 1090 1108 1090
query71 528 277 283 277
query72 6969 2642 2498 2498
query73 721 323 320 320
query74 7751 6551 6576 6551
query75 4188 2846 2818 2818
query76 5055 974 1018 974
query77 699 256 258 256
query78 10418 9522 9545 9522
query79 9880 502 509 502
query80 1580 386 371 371
query81 526 207 215 207
query82 849 204 208 204
query83 213 151 142 142
query84 282 76 75 75
query85 1436 322 314 314
query86 449 292 282 282
query87 3730 3507 3534 3507
query88 5046 2362 2337 2337
query89 512 349 356 349
query90 2014 176 172 172
query91 157 127 128 127
query92 61 46 49 46
query93 6940 493 469 469
query94 1179 177 176 176
query95 426 317 327 317
query96 604 270 260 260
query97 3053 2886 2872 2872
query98 225 208 210 208
query99 1125 764 746 746
Total cold run time: 312548 ms
Total hot run time: 179765 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1535004008
##########
be/src/exec/tablet_info.cpp:
##########
@@ -551,16 +504,84 @@ Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr,
part_key->second = column->size() - 1;
return Status::OK();
}
+// NOLINTEND(readability-function-size)
+
+Status VOlapTablePartitionParam::_create_partition_keys(const std::vector<TExprNode>& t_exprs,
+ BlockRow* part_key) {
+ for (int i = 0; i < t_exprs.size(); i++) {
+ RETURN_IF_ERROR(_create_partition_key(t_exprs[i], part_key, _partition_slot_locs[i]));
+ }
+ return Status::OK();
+}
+
+Status VOlapTablePartitionParam::generate_partition_from(const TOlapTablePartition& t_part,
+ VOlapTablePartition*& part_result) {
+ DCHECK(part_result == nullptr);
+ // here we set the default value of partition bounds first! if it doesn't have some key, it will be -1.
+ part_result = _obj_pool.add(new VOlapTablePartition(&_partition_block));
+ part_result->id = t_part.id;
+ part_result->is_mutable = t_part.is_mutable;
+ // only load_to_single_tablet = true will set load_tablet_idx
+ if (t_part.__isset.load_tablet_idx) {
+ part_result->load_tablet_idx = t_part.load_tablet_idx;
+ }
+
+ if (_is_in_partition) {
+ for (const auto& keys : t_part.in_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(
+ keys, &part_result->in_keys.emplace_back(&_partition_block, -1)));
+ }
+ if (t_part.__isset.is_default_partition && t_part.is_default_partition &&
+ _default_partition == nullptr) {
+ _default_partition = part_result;
+ }
+ } else { // range
+ if (t_part.__isset.start_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(t_part.start_keys, &part_result->start_key));
+ }
+ // we generate the right bound but not insert into partition map
+ if (t_part.__isset.end_keys) {
+ RETURN_IF_ERROR(_create_partition_keys(t_part.end_keys, &part_result->end_key));
+ }
+ }
+
+ part_result->num_buckets = t_part.num_buckets;
+ auto num_indexes = _schema->indexes().size();
+ if (t_part.indexes.size() != num_indexes) {
+ return Status::InternalError(
+ "number of partition's index is not equal with schema's"
+ ", num_part_indexes={}, num_schema_indexes={}",
+ t_part.indexes.size(), num_indexes);
+ }
+ part_result->indexes = t_part.indexes;
+ std::sort(part_result->indexes.begin(), part_result->indexes.end(),
+ [](const OlapTableIndexTablets& lhs, const OlapTableIndexTablets& rhs) {
+ return lhs.index_id < rhs.index_id;
+ });
+ // check index
+ for (int j = 0; j < num_indexes; ++j) {
+ if (part_result->indexes[j].index_id != _schema->indexes()[j]->index_id) {
+ std::stringstream ss;
Review Comment:
do not use `std::stringstream `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1528710514
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java:
##########
@@ -75,6 +76,10 @@ private void roundMicroSecond(int scale) {
this.second = localDateTime.getSecond();
this.microSecond -= 1000000;
}
+ if (checkRange() || checkDate()) {
+ // may fallback to legacy planner. make sure the behaviour of rounding is same.
Review Comment:
add some test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004368909
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2005838678
run p0 10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1528583202
##########
be/src/vec/sink/vrow_distribution.cpp:
##########
@@ -247,29 +256,34 @@ Status VRowDistribution::_generate_rows_distribution_for_auto_partition(
if (!_missing_map.empty()) {
// for missing partition keys, calc the missing partition and save in _partitions_need_create
- auto [part_ctxs, part_funcs] = _get_partition_function();
- auto funcs_size = part_funcs.size();
+ auto [part_ctxs, part_exprs] = _get_partition_function();
+ auto part_col_num = part_exprs.size();
+ // the two vectors are in column-first-order
std::vector<std::vector<std::string>> col_strs;
- col_strs.resize(funcs_size);
-
- for (int i = 0; i < funcs_size; ++i) {
- auto return_type = part_funcs[i]->data_type();
- // expose the data column
- vectorized::ColumnPtr range_left_col =
- block->get_by_position(partition_cols_idx[i]).column;
- if (const auto* nullable =
- check_and_get_column<vectorized::ColumnNullable>(*range_left_col)) {
- range_left_col = nullable->get_nested_column_ptr();
- return_type = assert_cast<const vectorized::DataTypeNullable*>(return_type.get())
- ->get_nested_type();
+ std::vector<const NullMap*> col_null_maps;
+ col_strs.resize(part_col_num);
+ col_null_maps.reserve(part_col_num);
Review Comment:
it has no perf diff here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1989117555
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004434493
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1528709862
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/PartitionExprUtil.java:
##########
@@ -124,15 +124,23 @@ public static Map<String, AddPartitionClause> getAddPartitionClauseFromPartition
FunctionIntervalInfo intervalInfo = getFunctionIntervalInfo(partitionExprs, partitionType);
Set<String> filterPartitionValues = new HashSet<String>();
- for (List<TStringLiteral> partitionValueList : partitionValues) {
+ for (List<TNullableStringLiteral> partitionValueList : partitionValues) {
PartitionKeyDesc partitionKeyDesc = null;
String partitionName = "p";
ArrayList<String> curPartitionValues = new ArrayList<>();
- for (TStringLiteral tStringLiteral : partitionValueList) {
- curPartitionValues.add(tStringLiteral.value);
+ for (TNullableStringLiteral tStringLiteral : partitionValueList) {
+ if (tStringLiteral.is_null) {
+ if (partitionType == PartitionType.RANGE) {
+ throw new AnalysisException("Can't create partition for NULL Range");
Review Comment:
add some test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1985012330
TeamCity be ut coverage result:
Function Coverage: 35.37% (8554/24186)
Line Coverage: 27.15% (69514/256054)
Region Coverage: 26.36% (36077/136847)
Branch Coverage: 23.21% (18438/79446)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0810c578b64f9151d25903ee31270e0e00ccf2cf_0810c578b64f9151d25903ee31270e0e00ccf2cf/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1979458160
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2015475097
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014133604
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 8f4a4904f509eb83193efb1bf29d516f9062ed9a with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 22.0 seconds inserted 10000000 Rows, about 454K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014351874
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a8b90b2d9f94602c21555f976383e2633d778c56 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.7 seconds inserted 10000000 Rows, about 460K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1710552345
##########
be/src/vec/common/assert_cast.h:
##########
@@ -20,25 +20,24 @@
#pragma once
-#include <string>
#include <type_traits>
-#include <typeindex>
#include <typeinfo>
#include "common/logging.h"
-#include "fmt/format.h"
#include "vec/common/demangle.h"
/** Perform static_cast in release build.
* Checks type by comparing typeid and throw an exception in debug build.
* The exact match of the type is checked. That is, cast to the ancestor will be unsuccessful.
*/
template <typename To, typename From>
-To assert_cast(From&& from) {
+PURE To assert_cast(From&& from) {
Review Comment:
why add `PURE` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1980294799
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature-WIP](partition) Suppor null range partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1980466661
TeamCity be ut coverage result:
Function Coverage: 35.41% (8559/24174)
Line Coverage: 27.16% (69486/255819)
Region Coverage: 26.37% (36037/136639)
Branch Coverage: 23.21% (18415/79342)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cfaf4663eea12e8bc9efe11149e685f91da78912_cfaf4663eea12e8bc9efe11149e685f91da78912/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1984325763
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1989299803
<details>
<summary>TPC-DS: <b>Total hot run time: 185981 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d, data reload: false
query1 915 346 349 346
query2 7377 1907 1850 1850
query3 6703 211 208 208
query4 24825 20883 20766 20766
query5 4311 471 533 471
query6 265 172 172 172
query7 4621 304 306 304
query8 233 167 169 167
query9 8409 2257 2279 2257
query10 464 237 259 237
query11 15070 14632 14287 14287
query12 130 94 87 87
query13 1652 432 456 432
query14 11091 10021 10420 10021
query15 260 208 194 194
query16 7712 278 268 268
query17 1875 568 553 553
query18 1959 271 272 271
query19 190 161 155 155
query20 93 89 83 83
query21 199 132 121 121
query22 4703 4386 4411 4386
query23 31372 30683 30702 30683
query24 10446 3093 3055 3055
query25 647 405 405 405
query26 1184 166 177 166
query27 2424 364 368 364
query28 6411 1846 1857 1846
query29 1017 607 613 607
query30 311 151 152 151
query31 933 744 720 720
query32 103 64 64 64
query33 767 264 256 256
query34 1052 488 499 488
query35 850 633 637 633
query36 960 831 818 818
query37 134 78 75 75
query38 3257 3171 3164 3164
query39 1529 1379 1379 1379
query40 214 119 115 115
query41 66 65 62 62
query42 112 107 101 101
query43 436 403 385 385
query44 1079 717 710 710
query45 275 257 257 257
query46 1047 817 771 771
query47 1606 1545 1557 1545
query48 446 358 365 358
query49 1129 347 334 334
query50 789 386 386 386
query51 6635 6582 6587 6582
query52 106 92 98 92
query53 357 285 283 283
query54 379 278 249 249
query55 94 85 85 85
query56 245 221 226 221
query57 1077 980 1008 980
query58 244 218 225 218
query59 2432 2281 2400 2281
query60 272 252 259 252
query61 113 111 113 111
query62 653 440 404 404
query63 307 282 322 282
query64 5225 3444 3254 3254
query65 3064 3000 2996 2996
query66 845 340 323 323
query67 14882 14323 14597 14323
query68 7000 572 575 572
query69 638 391 393 391
query70 1240 1106 1119 1106
query71 517 284 287 284
query72 6548 2665 2506 2506
query73 792 327 331 327
query74 6898 6534 6475 6475
query75 9648 8065 8226 8065
query76 4894 1102 1163 1102
query77 661 256 263 256
query78 10210 9529 9605 9529
query79 10193 524 515 515
query80 1783 381 371 371
query81 544 206 208 206
query82 796 207 197 197
query83 207 145 138 138
query84 279 77 79 77
query85 1506 336 317 317
query86 470 308 256 256
query87 3497 3238 3215 3215
query88 4385 2359 2351 2351
query89 557 361 362 361
query90 1944 179 175 175
query91 154 124 128 124
query92 57 49 49 49
query93 7214 540 526 526
query94 1227 188 186 186
query95 457 358 351 351
query96 618 263 262 262
query97 4426 4271 4265 4265
query98 229 205 202 202
query99 1156 757 754 754
Total cold run time: 301328 ms
Total hot run time: 185981 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-1989309431
<details>
<summary>ClickBench: <b>Total hot run time: 30.99 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39eae89849e935e37c255c5415dfcd9077136d9d, data reload: false
query1 0.04 0.03 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.10 0.10
query5 0.52 0.49 0.51
query6 1.38 0.63 0.67
query7 0.01 0.02 0.02
query8 0.03 0.03 0.03
query9 0.57 0.53 0.51
query10 0.57 0.55 0.58
query11 0.13 0.10 0.10
query12 0.12 0.10 0.10
query13 0.58 0.59 0.57
query14 0.74 0.75 0.75
query15 0.82 0.81 0.80
query16 0.38 0.37 0.38
query17 0.97 0.97 0.99
query18 0.26 0.25 0.25
query19 1.84 1.68 1.69
query20 0.02 0.01 0.01
query21 15.42 0.61 0.59
query22 3.24 3.80 2.56
query23 17.44 1.06 0.98
query24 2.22 0.56 0.52
query25 0.23 0.05 0.06
query26 0.19 0.15 0.14
query27 0.03 0.03 0.03
query28 11.95 0.85 0.84
query29 12.69 3.27 3.44
query30 0.63 0.58 0.53
query31 2.79 0.34 0.34
query32 3.39 0.44 0.43
query33 2.77 2.84 2.82
query34 15.48 4.30 4.31
query35 4.31 4.32 4.28
query36 1.09 1.02 1.01
query37 0.07 0.05 0.06
query38 0.04 0.02 0.03
query39 0.02 0.02 0.01
query40 0.19 0.16 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.27 s
Total hot run time: 30.99 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2014283945
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #31827:
URL: https://github.com/apache/doris/pull/31827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31827:
URL: https://github.com/apache/doris/pull/31827#issuecomment-2004408766
TeamCity be ut coverage result:
Function Coverage: 35.25% (8701/24687)
Line Coverage: 27.07% (71177/262924)
Region Coverage: 26.32% (36922/140276)
Branch Coverage: 23.24% (18884/81272)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c2e1b523d35a6eb2874554ae384d080ace102793_c2e1b523d35a6eb2874554ae384d080ace102793/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](partition) Support OLAP table null partition [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #31827:
URL: https://github.com/apache/doris/pull/31827#discussion_r1529669839
##########
be/src/exec/tablet_info.cpp:
##########
@@ -547,12 +536,13 @@ Status VOlapTablePartitionParam::_create_partition_key(const TExprNode& t_expr,
Status VOlapTablePartitionParam::add_partitions(
const std::vector<TOlapTablePartition>& partitions) {
for (const auto& t_part : partitions) {
- auto part = _obj_pool.add(new VOlapTablePartition(&_partition_block));
+ auto* part = _obj_pool.add(new VOlapTablePartition(&_partition_block));
part->id = t_part.id;
part->is_mutable = t_part.is_mutable;
- DCHECK(t_part.__isset.start_keys == t_part.__isset.end_keys &&
- t_part.__isset.start_keys != t_part.__isset.in_keys);
+ // we dont pass right keys when it's MAX_VALUE. so may dont have end_keys.
+ //
Review Comment:
remove the unless code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org