You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ziyanTOP (via GitHub)" <gi...@apache.org> on 2024/03/20 12:59:59 UTC
[PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
ziyanTOP opened a new pull request, #32569:
URL: https://github.com/apache/doris/pull/32569
## Proposed changes
Issue Number: close #32329
Avoid '24:00:00' time literal for datetimev1 column after handling by date function rewrite in nereids planner
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2015061272
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 912a6c0d9ed5f09b602a2d102011090864b530e2 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 20.7 seconds inserted 10000000 Rows, about 483K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2011160348
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2014926319
@ziyanTOP good job👍🏻, would you submit a fix pr for branch-2.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2009513678
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2014923821
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32569:
URL: https://github.com/apache/doris/pull/32569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on code in PR #32569:
URL: https://github.com/apache/doris/pull/32569#discussion_r1533203065
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java:
##########
@@ -458,10 +458,10 @@ public DateTimeLiteral toBeginOfTheDay() {
/**
* 2020-01-01
*
- * @return 2020-01-01 24:00:00
+ * @return 2020-01-01 23:59:59
*/
public DateTimeLiteral toEndOfTheDay() {
- return new DateTimeLiteral(year, month, day, 24, 0, 0);
+ return new DateTimeLiteral(year, month, day, 23, 59, 59);
Review Comment:
done @yiguolei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32569:
URL: https://github.com/apache/doris/pull/32569#discussion_r1532278555
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java:
##########
@@ -458,10 +458,10 @@ public DateTimeLiteral toBeginOfTheDay() {
/**
* 2020-01-01
*
- * @return 2020-01-01 24:00:00
+ * @return 2020-01-01 23:59:59
*/
public DateTimeLiteral toEndOfTheDay() {
- return new DateTimeLiteral(year, month, day, 24, 0, 0);
+ return new DateTimeLiteral(year, month, day, 23, 59, 59);
Review Comment:
please add a sql based regression test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2016324821
done @924060929
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2015056115
<details>
<summary>TPC-DS: <b>Total hot run time: 186861 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 912a6c0d9ed5f09b602a2d102011090864b530e2, data reload: false
query1 945 385 356 356
query2 7335 2081 2011 2011
query3 6704 213 222 213
query4 31840 21284 21387 21284
query5 4334 401 393 393
query6 271 182 177 177
query7 4633 290 293 290
query8 241 168 179 168
query9 9340 2289 2289 2289
query10 560 253 259 253
query11 14886 14568 14572 14568
query12 143 87 87 87
query13 1635 421 418 418
query14 11610 10907 11035 10907
query15 270 207 203 203
query16 8070 259 260 259
query17 1957 583 552 552
query18 2086 291 285 285
query19 279 157 159 157
query20 95 87 87 87
query21 203 128 128 128
query22 5055 4883 4777 4777
query23 33563 33050 32887 32887
query24 11618 2873 2824 2824
query25 665 389 394 389
query26 1844 161 154 154
query27 3077 353 359 353
query28 7663 1871 1857 1857
query29 1007 654 636 636
query30 306 147 149 147
query31 946 743 748 743
query32 88 59 58 58
query33 776 269 259 259
query34 1061 473 486 473
query35 840 608 603 603
query36 1008 891 880 880
query37 154 82 83 82
query38 3591 3452 3477 3452
query39 1511 1434 1406 1406
query40 303 114 113 113
query41 51 47 47 47
query42 104 93 95 93
query43 482 449 453 449
query44 1117 733 726 726
query45 294 259 271 259
query46 1115 725 688 688
query47 1925 1851 1866 1851
query48 443 357 353 353
query49 1229 339 363 339
query50 771 365 372 365
query51 6677 6597 6618 6597
query52 109 93 91 91
query53 354 277 284 277
query54 331 234 246 234
query55 84 79 76 76
query56 245 233 236 233
query57 1227 1130 1159 1130
query58 241 214 216 214
query59 2975 2670 2591 2591
query60 329 250 254 250
query61 116 112 115 112
query62 642 442 442 442
query63 307 280 280 280
query64 6874 3917 4087 3917
query65 3125 3023 3050 3023
query66 1414 378 353 353
query67 15570 14833 15115 14833
query68 10166 539 532 532
query69 689 383 374 374
query70 1350 1164 1101 1101
query71 549 274 271 271
query72 6677 2675 2542 2542
query73 1586 315 316 315
query74 7314 6678 6780 6678
query75 5172 2907 2916 2907
query76 5952 898 842 842
query77 637 280 261 261
query78 11048 10247 10113 10113
query79 11595 533 507 507
query80 1705 383 370 370
query81 499 225 217 217
query82 342 198 205 198
query83 216 141 142 141
query84 287 78 76 76
query85 1183 319 312 312
query86 357 320 305 305
query87 3733 3532 3550 3532
query88 5261 2287 2290 2287
query89 477 365 363 363
query90 2040 170 171 170
query91 163 136 136 136
query92 61 49 47 47
query93 6410 504 481 481
query94 1327 171 175 171
query95 452 328 331 328
query96 597 268 266 266
query97 3055 2882 2920 2882
query98 230 216 217 216
query99 1129 942 919 919
Total cold run time: 322923 ms
Total hot run time: 186861 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2014923868
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2014933122
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2015034802
<details>
<summary>TPC-H: <b>Total hot run time: 38474 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 912a6c0d9ed5f09b602a2d102011090864b530e2, data reload: false
------ Round 1 ----------------------------------
q1 17778 4906 4159 4159
q2 2113 162 168 162
q3 10884 1158 1247 1158
q4 10514 864 771 771
q5 7890 3067 3030 3030
q6 206 132 122 122
q7 1075 596 591 591
q8 9901 2047 2055 2047
q9 7390 6771 6695 6695
q10 9643 3488 3546 3488
q11 439 221 219 219
q12 389 196 192 192
q13 17809 2873 2879 2873
q14 239 210 214 210
q15 503 481 455 455
q16 491 377 371 371
q17 977 563 588 563
q18 7257 6557 6581 6557
q19 1556 1421 1507 1421
q20 551 277 257 257
q21 3532 2945 2837 2837
q22 339 296 296 296
Total cold run time: 111476 ms
Total hot run time: 38474 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4115 4098 4054 4054
q2 330 240 237 237
q3 2953 2883 2834 2834
q4 1879 1596 1581 1581
q5 5303 5356 5378 5356
q6 194 116 117 116
q7 2284 1885 1837 1837
q8 3151 3337 3274 3274
q9 8732 8715 8723 8715
q10 3777 3792 3781 3781
q11 541 448 439 439
q12 734 526 542 526
q13 16501 2874 2865 2865
q14 293 258 263 258
q15 517 470 472 470
q16 498 427 445 427
q17 1780 1548 1515 1515
q18 7677 7176 7075 7075
q19 1633 1583 1543 1543
q20 1920 1709 1717 1709
q21 4855 4826 4820 4820
q22 538 458 463 458
Total cold run time: 70205 ms
Total hot run time: 53890 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on PR #32569:
URL: https://github.com/apache/doris/pull/32569#issuecomment-2016264603
> @ziyanTOP good job👍🏻, would you submit a fix pr for branch-2.0?
ok, I will
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix] (Nereids) fix date function rewrite on datetimev1 column bug [doris]
Posted by "ziyanTOP (via GitHub)" <gi...@apache.org>.
ziyanTOP commented on code in PR #32569:
URL: https://github.com/apache/doris/pull/32569#discussion_r1533203065
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java:
##########
@@ -458,10 +458,10 @@ public DateTimeLiteral toBeginOfTheDay() {
/**
* 2020-01-01
*
- * @return 2020-01-01 24:00:00
+ * @return 2020-01-01 23:59:59
*/
public DateTimeLiteral toEndOfTheDay() {
- return new DateTimeLiteral(year, month, day, 24, 0, 0);
+ return new DateTimeLiteral(year, month, day, 23, 59, 59);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org