You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2023/11/24 17:09:10 UTC
[PR] [nereids] fix stats error when using dateTime type filter [doris]
xzj7019 opened a new pull request, #27571:
URL: https://github.com/apache/doris/pull/27571
## Proposed changes
Currently doris doesn't support datetime type filter stats estimation, but only for date type.
It will cause the filter using datetime type column with the same date and different time computing out a inaccurate selectivity and estimate a wrong row count, such as :
```
where o.book_time >= '2020-03-01 00:00:00.0' and o.book_time <= '2020-03-01 23:59:59.0';
```
This pr adds the datetime type(only support hh:mm:ss scale) filter estimation and improve the row count estimation for the above case.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [nereids] fix stats error when using dateTime type filter [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #27571:
URL: https://github.com/apache/doris/pull/27571#issuecomment-1825931628
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [nereids] fix stats error when using dateTime type filter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27571:
URL: https://github.com/apache/doris/pull/27571#issuecomment-1826169461
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [nereids] fix stats error when using dateTime type filter [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #27571:
URL: https://github.com/apache/doris/pull/27571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [nereids] fix stats error when using dateTime type filter [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27571:
URL: https://github.com/apache/doris/pull/27571#issuecomment-1826169447
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [nereids] fix stats error when using dateTime type filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27571:
URL: https://github.com/apache/doris/pull/27571#issuecomment-1825957901
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.03 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17098704373 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [nereids] fix stats error when using dateTime type filter [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27571:
URL: https://github.com/apache/doris/pull/27571#issuecomment-1825950742
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 4c2d3995d09333bb9efc2eed97b7401c14f640ec, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4939 4644 4645 4644
q2 353 144 159 144
q3 2037 2006 1890 1890
q4 1393 1231 1228 1228
q5 3970 3960 4018 3960
q6 246 134 136 134
q7 1428 881 890 881
q8 2787 2777 2771 2771
q9 9641 9500 9421 9421
q10 3462 3548 3509 3509
q11 369 247 248 247
q12 439 290 291 290
q13 4531 3791 3804 3791
q14 315 285 288 285
q15 606 539 533 533
q16 663 587 579 579
q17 1143 966 919 919
q18 7635 7213 7288 7213
q19 1681 1674 1701 1674
q20 566 300 281 281
q21 4350 3945 3931 3931
q22 478 380 380 380
Total cold run time: 53032 ms
Total hot run time: 48705 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4603 4585 4573 4573
q2 345 215 253 215
q3 4012 4002 3991 3991
q4 2664 2673 2665 2665
q5 9688 9711 9816 9711
q6 245 122 127 122
q7 3036 2450 2469 2450
q8 4483 4482 4484 4482
q9 12976 12857 12792 12792
q10 4064 4149 4180 4149
q11 793 669 641 641
q12 969 813 797 797
q13 4273 3593 3582 3582
q14 380 352 349 349
q15 583 528 518 518
q16 735 696 661 661
q17 3844 3782 3896 3782
q18 9435 8931 8865 8865
q19 1799 1817 1750 1750
q20 2388 2072 2037 2037
q21 8807 8670 8550 8550
q22 922 803 838 803
Total cold run time: 81044 ms
Total hot run time: 77485 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org