You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "dataroaring (via GitHub)" <gi...@apache.org> on 2024/07/03 09:35:13 UTC
[PR] [fix](meta) do not serialize nametotable map [doris]
dataroaring opened a new pull request, #37222:
URL: https://github.com/apache/doris/pull/37222
Gson can not handle size beyonds 2GB.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205757965
<details>
<summary>TPC-DS: <b>Total hot run time: 173564 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7973108f9b41b85383b772884be6e9ec5a4857a, data reload: false
query1 916 368 364 364
query2 6444 2527 2272 2272
query3 6645 201 210 201
query4 19188 17446 17115 17115
query5 3688 468 463 463
query6 257 178 165 165
query7 4594 300 285 285
query8 315 288 294 288
query9 8465 2381 2357 2357
query10 560 296 289 289
query11 10683 9908 9925 9908
query12 122 84 80 80
query13 1646 379 372 372
query14 10276 7553 7556 7553
query15 240 191 191 191
query16 8021 292 267 267
query17 1886 554 538 538
query18 2034 280 277 277
query19 200 154 154 154
query20 99 81 87 81
query21 215 135 142 135
query22 4265 4019 3998 3998
query23 33967 33670 33609 33609
query24 10964 2879 2907 2879
query25 601 421 390 390
query26 710 165 162 162
query27 2255 323 322 322
query28 5934 2141 2158 2141
query29 888 668 648 648
query30 245 159 156 156
query31 1029 787 763 763
query32 104 57 65 57
query33 778 323 300 300
query34 995 472 496 472
query35 751 642 615 615
query36 1143 1001 967 967
query37 137 80 79 79
query38 2974 2860 2894 2860
query39 886 860 900 860
query40 201 129 124 124
query41 52 52 56 52
query42 103 99 99 99
query43 566 549 532 532
query44 1160 758 728 728
query45 199 163 163 163
query46 1078 691 702 691
query47 1825 1773 1770 1770
query48 363 305 300 300
query49 836 416 406 406
query50 767 375 383 375
query51 6799 6713 6837 6713
query52 102 89 91 89
query53 369 286 294 286
query54 920 444 441 441
query55 73 74 70 70
query56 275 253 259 253
query57 1143 1066 1049 1049
query58 245 234 243 234
query59 3268 3168 3111 3111
query60 298 275 278 275
query61 89 103 86 86
query62 596 457 428 428
query63 321 286 288 286
query64 8509 2251 1731 1731
query65 3143 3089 3108 3089
query66 740 315 323 315
query67 15490 15132 14962 14962
query68 6253 537 542 537
query69 764 465 333 333
query70 1240 1169 1149 1149
query71 521 277 266 266
query72 9107 5570 4845 4845
query73 791 324 325 324
query74 5883 5475 5524 5475
query75 5079 2677 2658 2658
query76 4595 892 948 892
query77 788 316 305 305
query78 10444 9943 9835 9835
query79 6390 511 552 511
query80 934 468 463 463
query81 538 214 217 214
query82 768 112 104 104
query83 338 173 166 166
query84 267 93 91 91
query85 1257 274 266 266
query86 407 303 296 296
query87 3313 3099 3080 3080
query88 4135 2435 2431 2431
query89 497 379 383 379
query90 1978 181 189 181
query91 166 99 100 99
query92 57 46 49 46
query93 5328 521 516 516
query94 1203 190 187 187
query95 408 315 310 310
query96 610 274 264 264
query97 3161 3021 3025 3021
query98 218 206 187 187
query99 1146 842 845 842
Total cold run time: 283234 ms
Total hot run time: 173564 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205572984
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #37222:
URL: https://github.com/apache/doris/pull/37222#discussion_r1663905511
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java:
##########
@@ -650,6 +676,10 @@ public String getSignature(int signatureVersion) {
public void write(DataOutput out) throws IOException {
discardHudiTable();
Text.writeString(out, GsonUtils.GSON.toJson(this));
+ if (FeMetaVersion.VERSION_138 <= Env.getCurrentEnvJournalVersion()) {
+ writeTables(out);
+ return;
+ }
Review Comment:
```suggestion
writeTables(out);
return;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2206073938
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205629335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "w41ter (via GitHub)" <gi...@apache.org>.
w41ter commented on code in PR #37222:
URL: https://github.com/apache/doris/pull/37222#discussion_r1663920754
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java:
##########
@@ -650,6 +676,8 @@ public String getSignature(int signatureVersion) {
public void write(DataOutput out) throws IOException {
discardHudiTable();
Text.writeString(out, GsonUtils.GSON.toJson(this));
+ writeTables(out);
+ return;
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #37222:
URL: https://github.com/apache/doris/pull/37222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205768208
<details>
<summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7973108f9b41b85383b772884be6e9ec5a4857a, data reload: false
query1 0.04 0.03 0.02
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.69 0.08 0.08
query5 0.51 0.49 0.50
query6 1.14 0.72 0.72
query7 0.02 0.01 0.02
query8 0.04 0.04 0.04
query9 0.54 0.48 0.49
query10 0.54 0.53 0.54
query11 0.16 0.11 0.11
query12 0.15 0.12 0.13
query13 0.60 0.58 0.59
query14 0.76 0.77 0.82
query15 0.85 0.81 0.81
query16 0.37 0.35 0.37
query17 0.96 1.02 0.99
query18 0.23 0.23 0.25
query19 1.82 1.76 1.67
query20 0.01 0.01 0.01
query21 15.41 0.78 0.65
query22 4.32 6.89 2.05
query23 18.28 1.33 1.24
query24 2.06 0.26 0.22
query25 0.14 0.08 0.08
query26 0.27 0.18 0.17
query27 0.08 0.09 0.08
query28 13.24 1.00 1.00
query29 12.60 3.24 3.22
query30 0.25 0.07 0.06
query31 2.85 0.38 0.38
query32 3.30 0.48 0.47
query33 2.90 2.93 2.88
query34 17.13 4.46 4.42
query35 4.47 4.53 4.47
query36 0.65 0.46 0.47
query37 0.19 0.16 0.16
query38 0.16 0.15 0.15
query39 0.05 0.03 0.04
query40 0.17 0.14 0.15
query41 0.09 0.06 0.05
query42 0.06 0.06 0.04
query43 0.04 0.04 0.04
Total cold run time: 109.44 s
Total hot run time: 30.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205580955
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205600706
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205612525
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205612655
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](meta) do not serialize nametotable map [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37222:
URL: https://github.com/apache/doris/pull/37222#issuecomment-2205736942
<details>
<summary>TPC-H: <b>Total hot run time: 39863 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7973108f9b41b85383b772884be6e9ec5a4857a, data reload: false
------ Round 1 ----------------------------------
q1 17627 4401 4261 4261
q2 2018 187 181 181
q3 10485 1255 1191 1191
q4 10186 761 747 747
q5 7455 2652 2632 2632
q6 220 136 138 136
q7 956 598 613 598
q8 9223 2065 2051 2051
q9 8762 6448 6452 6448
q10 8903 3767 3782 3767
q11 478 238 245 238
q12 409 247 241 241
q13 17840 3014 3018 3014
q14 272 222 235 222
q15 537 494 494 494
q16 536 392 383 383
q17 961 676 652 652
q18 8124 7393 7363 7363
q19 7267 1565 1429 1429
q20 677 323 317 317
q21 4955 3161 3285 3161
q22 399 337 341 337
Total cold run time: 118290 ms
Total hot run time: 39863 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4390 4187 4252 4187
q2 374 266 257 257
q3 2990 2930 2971 2930
q4 1994 1726 1730 1726
q5 5574 5468 5453 5453
q6 227 136 142 136
q7 2262 1884 1856 1856
q8 3230 3396 3407 3396
q9 8652 8695 8767 8695
q10 4106 3792 3747 3747
q11 622 485 503 485
q12 820 695 647 647
q13 16154 3222 3236 3222
q14 308 269 283 269
q15 512 498 481 481
q16 478 440 427 427
q17 1793 1515 1514 1514
q18 8149 7967 7753 7753
q19 1794 1796 1533 1533
q20 2163 1860 1883 1860
q21 5102 4911 4777 4777
q22 635 599 564 564
Total cold run time: 72329 ms
Total hot run time: 55915 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org