You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "cambyzju (via GitHub)" <gi...@apache.org> on 2025/12/24 12:00:45 UTC
[PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
cambyzju opened a new pull request, #59340:
URL: https://github.com/apache/doris/pull/59340
### What problem does this PR solve?
**Usage Example:**
```
> select * from test_pk@ro where user_id=9820660;
+---------+----------+
| user_id | behavior |
+---------+----------+
| 9820660 | hello |
+---------+----------+
```
- When there are overlapping key ranges in Paimon Primary Key table, we cannot use native reader:
```
> explain verbose select * from test_pk where user_id=9820660;
+------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| user_id[#0] |
| behavior[#1] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCAL |
| |
| 1:VEXCHANGE |
| offset: 0 |
| distribute expr lists: |
| tuple ids: 0 |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: RANDOM |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 01 |
| UNPARTITIONED |
| |
| 0:VPAIMON_SCAN_NODE(79) |
| table: paimon_on_s3.db_paimon.test_pk |
| predicates: (user_id[#0] = 9820660) |
| inputSplitNum=1, totalFileSize=0, scanRanges=1 |
| partition=1/0 |
| backends: |
| 1754986988417 |
| /data-0891318b-4e3b-42b5-b85c-3bc8e9b8cc45-0.parquet start: 0 length: 0 |
| cardinality=1001, numNodes=1 |
| pushdown agg=NONE |
| paimonNativeReadSplits=0/1 |
| predicatesFromPaimon: |
| Equal(user_id, 9820660) |
| PaimonTable: test_pk |
| PaimonSplitStats: |
| SplitStat [type=JNI, rowCount=1001, mergedRowCount=NONE, rawFileConvertable=false, hasDeletionVector=false] |
| tuple ids: 0 |
| |
| Tuples: |
| TupleDescriptor{id=0, tbl=test_pk} |
| SlotDescriptor{id=0, col=user_id, colUniqueId=0, type=bigint, nullable=true, isAutoIncrement=false, subColPath=null} |
| SlotDescriptor{id=1, col=behavior, colUniqueId=1, type=text, nullable=true, isAutoIncrement=false, subColPath=null} |
| |
| |
| |
| |
| ========== STATISTICS ========== |
| planed with unknown column statistics |
+------------------------------------------------------------------------------------------------------------------------+
```
- But if we try to read `ro` table, we could use the Native reader with some data latency:
```
> explain verbose select * from test_pk@ro where user_id=9820660;
+--------------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+--------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| user_id[#0] |
| behavior[#1] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCAL |
| |
| 1:VEXCHANGE |
| offset: 0 |
| distribute expr lists: |
| tuple ids: 0 |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: RANDOM |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 01 |
| UNPARTITIONED |
| |
| 0:VPAIMON_SCAN_NODE(79) |
| table: paimon_on_s3.db_paimon.test_pk |
| predicates: (user_id[#0] = 9820660) |
| inputSplitNum=1, totalFileSize=519490, scanRanges=1 |
| partition=1/0 |
| backends: |
| 1754986988417 |
| s3://warehouse/wh/db_paimon.db/test_pk/bucket-0/data-1ae9ee1e-e6aa-4594-9506-75b6b6ad3820-0.parquet start: 0 length: 519490 |
| cardinality=1001, numNodes=1 |
| pushdown agg=NONE |
| paimonNativeReadSplits=1/1 |
| predicatesFromPaimon: |
| Equal(user_id, 9820660) |
| PaimonTable: test_pk$ro |
| PaimonSplitStats: |
| SplitStat [type=NATIVE, rowCount=1000, mergedRowCount=NONE, rawFileConvertable=true, hasDeletionVector=false] |
| tuple ids: 0 |
| |
| Tuples: |
| TupleDescriptor{id=0, tbl=test_pk} |
| SlotDescriptor{id=0, col=user_id, colUniqueId=0, type=bigint, nullable=true, isAutoIncrement=false, subColPath=null} |
| SlotDescriptor{id=1, col=behavior, colUniqueId=1, type=text, nullable=true, isAutoIncrement=false, subColPath=null} |
| |
| |
| |
| |
| ========== STATISTICS ========== |
| planed with unknown column statistics |
+--------------------------------------------------------------------------------------------------------------------------------------+
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3692022893
<details>
<summary>ClickBench: <b>Total hot run time: 27.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1a09485c65826631b1f0971441d45ccc106a758, data reload: false
query1 0.05 0.04 0.05
query2 0.10 0.05 0.05
query3 0.26 0.08 0.08
query4 1.61 0.12 0.11
query5 0.27 0.25 0.24
query6 1.18 0.65 0.65
query7 0.03 0.03 0.02
query8 0.05 0.04 0.04
query9 0.56 0.51 0.50
query10 0.55 0.56 0.55
query11 0.16 0.10 0.10
query12 0.15 0.12 0.12
query13 0.61 0.60 0.60
query14 0.98 0.98 0.97
query15 0.83 0.80 0.81
query16 0.39 0.39 0.39
query17 1.02 1.00 1.03
query18 0.23 0.22 0.21
query19 1.93 1.88 1.80
query20 0.02 0.01 0.01
query21 15.44 0.29 0.14
query22 4.66 0.05 0.05
query23 15.96 0.28 0.11
query24 1.80 0.58 0.20
query25 0.08 0.08 0.05
query26 0.14 0.13 0.13
query27 0.07 0.06 0.08
query28 4.53 1.24 1.03
query29 12.62 3.92 3.26
query30 0.27 0.14 0.11
query31 2.81 0.62 0.40
query32 3.23 0.56 0.44
query33 2.96 3.00 3.00
query34 17.03 5.15 4.54
query35 4.60 4.55 4.59
query36 0.66 0.50 0.49
query37 0.11 0.07 0.07
query38 0.08 0.04 0.04
query39 0.05 0.04 0.03
query40 0.16 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.05 0.03 0.04
Total cold run time: 98.42 s
Total hot run time: 27.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "cambyzju (via GitHub)" <gi...@apache.org>.
cambyzju commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3691969021
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3691551362
<details>
<summary>TPC-DS: <b>Total hot run time: 180031 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3d143133d46ea81899bff8cebfac0b09985296c, data reload: false
query5 4429 606 475 475
query6 366 245 259 245
query7 4229 477 274 274
query8 309 260 257 257
query9 8774 2546 2553 2546
query10 529 394 333 333
query11 15637 14936 14903 14903
query12 177 123 116 116
query13 1276 521 395 395
query14 5769 3054 2788 2788
query14_1 2794 2706 2705 2705
query15 237 202 183 183
query16 848 457 492 457
query17 1178 738 620 620
query18 2446 450 359 359
query19 236 234 215 215
query20 123 120 118 118
query21 227 146 123 123
query22 3905 3845 3948 3845
query23 16825 16191 16086 16086
query23_1 16057 16075 15965 15965
query24 7353 1683 1241 1241
query24_1 1250 1287 1280 1280
query25 597 513 468 468
query26 1272 273 168 168
query27 2754 473 307 307
query28 4488 2131 2106 2106
query29 861 580 478 478
query30 318 248 216 216
query31 811 751 649 649
query32 78 72 69 69
query33 533 340 289 289
query34 912 953 541 541
query35 767 813 714 714
query36 858 909 820 820
query37 138 93 81 81
query38 2981 3065 3037 3037
query39 770 757 736 736
query39_1 701 719 709 709
query40 228 141 125 125
query41 69 67 64 64
query42 114 111 111 111
query43 439 440 402 402
query44 1329 753 736 736
query45 190 190 183 183
query46 891 990 616 616
query47 1663 1724 1633 1633
query48 334 333 250 250
query49 621 441 357 357
query50 660 296 217 217
query51 3827 3828 3786 3786
query52 110 109 100 100
query53 322 358 303 303
query54 287 251 242 242
query55 77 72 73 72
query56 294 296 294 294
query57 1170 1143 1101 1101
query58 276 255 247 247
query59 2397 2484 2356 2356
query60 326 311 296 296
query61 164 162 165 162
query62 771 697 687 687
query63 333 297 313 297
query64 5128 1295 992 992
query65 4006 3946 3994 3946
query66 1512 443 331 331
query67 15415 15018 14907 14907
query68 8409 1000 721 721
query69 516 351 306 306
query70 1072 996 948 948
query71 381 329 285 285
query72 6087 5041 4988 4988
query73 679 598 311 311
query74 8933 8983 8813 8813
query75 3267 3151 2832 2832
query76 3901 1156 758 758
query77 626 394 307 307
query78 9419 9409 8876 8876
query79 1640 871 619 619
query80 725 664 550 550
query81 505 273 233 233
query82 484 127 101 101
query83 297 262 243 243
query84 294 119 102 102
query85 971 508 452 452
query86 344 311 270 270
query87 3180 3218 3088 3088
query88 4174 2295 2293 2293
query89 487 426 392 392
query90 2168 158 160 158
query91 170 169 147 147
query92 81 69 68 68
query93 1443 907 557 557
query94 477 319 295 295
query95 581 386 316 316
query96 600 470 208 208
query97 2314 2320 2225 2225
query98 209 192 196 192
query99 1379 1386 1322 1322
Total cold run time: 262807 ms
Total hot run time: 180031 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3689600790
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3889426524
supported in #60505 60556
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3692197410
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/18)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/59340_e1a09485c65826631b1f0971441d45ccc106a758_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/59340_e1a09485c65826631b1f0971441d45ccc106a758_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "cambyzju (via GitHub)" <gi...@apache.org>.
cambyzju commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3691463843
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3691554494
<details>
<summary>ClickBench: <b>Total hot run time: 27.61 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3d143133d46ea81899bff8cebfac0b09985296c, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.04
query3 0.26 0.09 0.09
query4 1.60 0.11 0.11
query5 0.28 0.26 0.26
query6 1.18 0.65 0.63
query7 0.03 0.03 0.03
query8 0.05 0.04 0.04
query9 0.56 0.49 0.51
query10 0.55 0.56 0.53
query11 0.16 0.12 0.11
query12 0.14 0.12 0.11
query13 0.60 0.60 0.61
query14 0.99 0.97 0.98
query15 0.81 0.79 0.81
query16 0.39 0.38 0.39
query17 1.05 1.04 1.05
query18 0.23 0.22 0.22
query19 1.87 1.81 1.84
query20 0.02 0.01 0.02
query21 15.48 0.29 0.14
query22 6.04 0.05 0.05
query23 15.94 0.28 0.10
query24 1.52 0.55 0.56
query25 0.11 0.10 0.08
query26 0.14 0.14 0.13
query27 0.08 0.05 0.06
query28 4.90 1.22 1.02
query29 12.65 4.00 3.25
query30 0.29 0.14 0.12
query31 2.82 0.62 0.39
query32 3.23 0.55 0.46
query33 3.05 3.03 3.10
query34 16.84 5.17 4.52
query35 4.57 4.58 4.61
query36 0.67 0.51 0.48
query37 0.12 0.07 0.06
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.19 0.14 0.14
query41 0.08 0.03 0.03
query42 0.05 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 99.86 s
Total hot run time: 27.61 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman closed pull request #59340: [feature](paimon) support read paimon ro (read-optimized) system table
URL: https://github.com/apache/doris/pull/59340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3691668911
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/18)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/59340_e3d143133d46ea81899bff8cebfac0b09985296c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/59340_e3d143133d46ea81899bff8cebfac0b09985296c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3692002482
<details>
<summary>TPC-H: <b>Total hot run time: 35864 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1a09485c65826631b1f0971441d45ccc106a758, data reload: false
------ Round 1 ----------------------------------
q1 17611 4219 4044 4044
q2 2018 361 243 243
q3 10189 1306 729 729
q4 10209 810 311 311
q5 7533 2140 1879 1879
q6 185 170 139 139
q7 1007 848 717 717
q8 9352 1387 1146 1146
q9 6923 5327 5339 5327
q10 6850 2380 1994 1994
q11 545 342 325 325
q12 661 715 585 585
q13 17784 3651 2980 2980
q14 294 294 271 271
q15 583 516 521 516
q16 690 684 621 621
q17 697 833 514 514
q18 7735 7483 7850 7483
q19 1218 1058 616 616
q20 423 369 254 254
q21 4534 4238 4136 4136
q22 1147 1070 1034 1034
Total cold run time: 108188 ms
Total hot run time: 35864 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4606 4204 4297 4204
q2 312 440 327 327
q3 2340 3016 2459 2459
q4 1397 1918 1456 1456
q5 4601 4373 4517 4373
q6 217 169 129 129
q7 2028 1873 1815 1815
q8 2853 2473 2472 2472
q9 7481 7488 7298 7298
q10 2927 3085 2673 2673
q11 601 518 509 509
q12 664 689 576 576
q13 3280 3615 3018 3018
q14 265 287 256 256
q15 535 492 488 488
q16 602 646 605 605
q17 1119 1405 1332 1332
q18 7376 7055 7210 7055
q19 827 774 792 774
q20 1914 1961 1851 1851
q21 4602 4391 4194 4194
q22 1049 1022 986 986
Total cold run time: 51596 ms
Total hot run time: 48850 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3691541291
<details>
<summary>TPC-H: <b>Total hot run time: 35565 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3d143133d46ea81899bff8cebfac0b09985296c, data reload: false
------ Round 1 ----------------------------------
q1 17652 4208 4051 4051
q2 2023 353 237 237
q3 10178 1363 780 780
q4 10236 944 324 324
q5 7530 2170 2087 2087
q6 192 175 142 142
q7 1023 846 722 722
q8 9380 1474 1212 1212
q9 7188 5395 5365 5365
q10 6846 2422 1993 1993
q11 541 341 314 314
q12 689 753 564 564
q13 17785 3737 3068 3068
q14 289 312 267 267
q15 606 520 509 509
q16 733 686 646 646
q17 722 866 489 489
q18 7514 7105 7122 7105
q19 1102 989 610 610
q20 409 377 248 248
q21 4366 3896 3931 3896
q22 1073 1008 936 936
Total cold run time: 108077 ms
Total hot run time: 35565 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4139 4064 4025 4025
q2 348 395 320 320
q3 2159 2669 2319 2319
q4 1359 1774 1292 1292
q5 4324 4854 4731 4731
q6 231 180 142 142
q7 2117 1960 1912 1912
q8 3005 2632 2586 2586
q9 7835 7704 7384 7384
q10 3115 3325 2863 2863
q11 620 535 511 511
q12 687 755 604 604
q13 3589 3981 3335 3335
q14 301 302 301 301
q15 572 519 505 505
q16 656 702 655 655
q17 1223 1472 1514 1472
q18 8080 7656 7583 7583
q19 925 934 944 934
q20 2015 2055 1922 1922
q21 4941 4654 4260 4260
q22 1076 994 980 980
Total cold run time: 53317 ms
Total hot run time: 50636 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59340:
URL: https://github.com/apache/doris/pull/59340#issuecomment-3692014613
<details>
<summary>TPC-DS: <b>Total hot run time: 180298 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1a09485c65826631b1f0971441d45ccc106a758, data reload: false
query5 5180 571 444 444
query6 326 229 213 213
query7 4222 454 275 275
query8 299 258 230 230
query9 8743 2538 2516 2516
query10 545 352 321 321
query11 15377 14964 14807 14807
query12 173 123 122 122
query13 1272 473 400 400
query14 6138 2952 2686 2686
query14_1 2602 2572 2578 2572
query15 215 191 173 173
query16 799 472 443 443
query17 1078 693 565 565
query18 2457 429 329 329
query19 226 226 200 200
query20 121 116 112 112
query21 225 139 124 124
query22 4086 4063 4123 4063
query23 16617 16377 16227 16227
query23_1 16158 16438 16449 16438
query24 7367 1669 1256 1256
query24_1 1237 1237 1216 1216
query25 572 498 441 441
query26 1247 271 167 167
query27 2734 471 304 304
query28 4483 2106 2131 2106
query29 827 566 481 481
query30 318 245 221 221
query31 815 701 633 633
query32 79 76 68 68
query33 570 336 319 319
query34 887 891 538 538
query35 780 801 721 721
query36 849 935 854 854
query37 130 91 81 81
query38 3012 3074 2962 2962
query39 767 750 763 750
query39_1 711 742 704 704
query40 230 149 129 129
query41 76 70 66 66
query42 112 107 106 106
query43 418 446 405 405
query44 1321 753 741 741
query45 191 191 185 185
query46 884 971 608 608
query47 1706 1778 1670 1670
query48 327 326 245 245
query49 644 450 364 364
query50 661 300 227 227
query51 3809 3861 3807 3807
query52 114 109 99 99
query53 321 357 299 299
query54 300 270 259 259
query55 76 78 74 74
query56 297 310 321 310
query57 1193 1174 1104 1104
query58 281 256 266 256
query59 2463 2462 2473 2462
query60 365 308 296 296
query61 166 150 163 150
query62 710 711 640 640
query63 326 286 295 286
query64 4948 1267 978 978
query65 4028 3926 3967 3926
query66 1448 471 323 323
query67 15355 15255 14928 14928
query68 8428 993 698 698
query69 503 344 318 318
query70 1089 1030 975 975
query71 363 311 287 287
query72 6118 4984 5141 4984
query73 705 651 306 306
query74 8959 8917 8739 8739
query75 3195 3164 2768 2768
query76 3870 1140 737 737
query77 517 394 292 292
query78 9512 9642 8901 8901
query79 1755 836 606 606
query80 709 653 558 558
query81 514 271 233 233
query82 195 130 104 104
query83 260 258 243 243
query84 258 122 110 110
query85 932 500 449 449
query86 378 313 286 286
query87 3205 3154 3225 3154
query88 3244 2276 2264 2264
query89 457 435 397 397
query90 2165 159 159 159
query91 174 167 143 143
query92 75 68 65 65
query93 1388 885 550 550
query94 464 281 276 276
query95 567 373 302 302
query96 588 463 208 208
query97 2241 2340 2255 2255
query98 219 194 195 194
query99 1325 1405 1291 1291
Total cold run time: 261201 ms
Total hot run time: 180298 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](paimon) support read paimon ro (read-optimized) system table [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #59340:
URL: https://github.com/apache/doris/pull/59340#discussion_r2647100927
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TableScanParams.java:
##########
@@ -29,24 +29,32 @@ public class TableScanParams {
public static final String INCREMENTAL_READ = "incr";
public static final String BRANCH = "branch";
public static final String TAG = "tag";
+ // for paimon read-optimized system table
+ public static final String RO = "ro";
private static final ImmutableSet<String> VALID_PARAM_TYPES = ImmutableSet.of(
INCREMENTAL_READ,
BRANCH,
- TAG);
+ TAG,
+ RO);
private final String paramType;
- // There are two ways to pass parameters to a function.
+ // There are three ways to pass parameters to a function.
// - One is in map form, where the data is stored in `mapParams`.
// such as: @func_name('param1'='value1', 'param2'='value2', 'param3'='value3')
- // - The other is in list form, where the data is stored in `listParams`.
+ // - Another other one is in list form, where the data is stored in `listParams`.
// such as: `listParams` is used for @func_name('value1', 'value2', 'value3')
+ // - The third one is in short form, now only used for paimon 'ro' table.
+ // such as: @ro
private final Map<String, String> mapParams;
private final List<String> listParams;
private void validate() {
if (!VALID_PARAM_TYPES.contains(paramType)) {
throw new IllegalArgumentException("Invalid param type: " + paramType);
}
+ if (isRo() && (!mapParams.isEmpty() || !listParams.isEmpty())) {
+ throw new IllegalArgumentException("Please use '@ro' for Paimon read-optimized system table");
Review Comment:
The error message says "Please use '@ro' for Paimon read-optimized system table", but this validation is triggered when RO is used WITH parameters (like @ro('param'='value')). The message could be clearer by specifying that RO should be used without parameters, for example: "The '@ro' parameter for Paimon read-optimized tables should be used without arguments".
```suggestion
throw new IllegalArgumentException("The '@ro' parameter for Paimon read-optimized tables must be used without arguments");
```
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/TableScanParams.java:
##########
@@ -29,24 +29,32 @@ public class TableScanParams {
public static final String INCREMENTAL_READ = "incr";
public static final String BRANCH = "branch";
public static final String TAG = "tag";
+ // for paimon read-optimized system table
+ public static final String RO = "ro";
private static final ImmutableSet<String> VALID_PARAM_TYPES = ImmutableSet.of(
INCREMENTAL_READ,
BRANCH,
- TAG);
+ TAG,
+ RO);
private final String paramType;
- // There are two ways to pass parameters to a function.
+ // There are three ways to pass parameters to a function.
// - One is in map form, where the data is stored in `mapParams`.
// such as: @func_name('param1'='value1', 'param2'='value2', 'param3'='value3')
- // - The other is in list form, where the data is stored in `listParams`.
+ // - Another other one is in list form, where the data is stored in `listParams`.
Review Comment:
The phrase "Another other one" is grammatically incorrect. It should be either "Another one" or "The other one" to maintain grammatical clarity.
```suggestion
// - Another one is in list form, where the data is stored in `listParams`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org