You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wuwenchi (via GitHub)" <gi...@apache.org> on 2024/04/19 06:24:53 UTC
[PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
wuwenchi opened a new pull request, #33883:
URL: https://github.com/apache/doris/pull/33883
## Proposed changes
Use `MetaIdGenerator` instead of the `TransactionIdGenerator`. Because the `TransactionIdGenerator` will record `OP_SAVE_TRANSACTION_ID` data in the editlog, when restoring the data, it will restore the data to `GlobalTransactionMgr`, but `GlobalTransactionMgr` is not supported in `Cloud` mode.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2066235672
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2065846062
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2066302889
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 0f9deb79579005bbac6f4b8999a97e635a5a7d34 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.2 seconds inserted 10000000 Rows, about 704K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2065845584
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2066095207
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2065843385
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #33883:
URL: https://github.com/apache/doris/pull/33883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2065846770
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2066298309
<details>
<summary>ClickBench: <b>Total hot run time: 30.32 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f9deb79579005bbac6f4b8999a97e635a5a7d34, data reload: false
query1 0.03 0.03 0.03
query2 0.09 0.04 0.04
query3 0.23 0.06 0.06
query4 1.68 0.07 0.06
query5 0.48 0.53 0.50
query6 1.46 0.72 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.53 0.51 0.50
query10 0.54 0.55 0.55
query11 0.16 0.11 0.12
query12 0.15 0.12 0.12
query13 0.60 0.58 0.58
query14 0.74 0.78 0.78
query15 0.83 0.82 0.80
query16 0.37 0.34 0.36
query17 0.95 0.94 1.00
query18 0.20 0.24 0.24
query19 1.87 1.79 1.64
query20 0.01 0.01 0.01
query21 15.40 0.66 0.65
query22 3.85 7.92 1.98
query23 18.34 1.40 1.28
query24 1.37 0.39 0.23
query25 0.15 0.09 0.09
query26 0.27 0.16 0.16
query27 0.08 0.07 0.09
query28 13.37 1.02 0.98
query29 12.56 3.24 3.26
query30 0.25 0.06 0.05
query31 2.86 0.38 0.39
query32 3.28 0.47 0.47
query33 2.82 2.87 2.80
query34 17.07 4.46 4.41
query35 4.51 4.51 4.42
query36 0.64 0.46 0.46
query37 0.19 0.15 0.15
query38 0.15 0.15 0.15
query39 0.04 0.03 0.04
query40 0.18 0.14 0.14
query41 0.09 0.05 0.04
query42 0.05 0.05 0.04
query43 0.04 0.04 0.03
Total cold run time: 108.55 s
Total hot run time: 30.32 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2066270212
<details>
<summary>TPC-H: <b>Total hot run time: 38567 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f9deb79579005bbac6f4b8999a97e635a5a7d34, data reload: false
------ Round 1 ----------------------------------
q1 18078 4539 4266 4266
q2 2014 193 193 193
q3 10548 1162 1153 1153
q4 10262 803 822 803
q5 7513 2678 2629 2629
q6 218 131 129 129
q7 1009 635 587 587
q8 9223 2052 2026 2026
q9 7318 6624 6544 6544
q10 8427 3596 3558 3558
q11 453 238 247 238
q12 389 214 214 214
q13 17774 2895 2931 2895
q14 257 236 238 236
q15 532 491 492 491
q16 540 384 376 376
q17 961 635 646 635
q18 7546 6812 6762 6762
q19 6733 1523 1497 1497
q20 655 316 297 297
q21 3435 2738 2838 2738
q22 368 318 300 300
Total cold run time: 114253 ms
Total hot run time: 38567 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4328 4202 4233 4202
q2 369 277 273 273
q3 3027 2766 2779 2766
q4 1841 1599 1671 1599
q5 5379 5340 5301 5301
q6 210 122 126 122
q7 2250 1903 1919 1903
q8 3223 3331 3351 3331
q9 8658 8585 8648 8585
q10 3900 3706 3748 3706
q11 581 478 479 478
q12 751 587 578 578
q13 16305 2944 2934 2934
q14 301 284 271 271
q15 510 475 478 475
q16 472 409 430 409
q17 1778 1487 1488 1487
q18 7671 7658 7508 7508
q19 1630 1558 1536 1536
q20 1960 1784 1732 1732
q21 4921 4851 4877 4851
q22 531 467 475 467
Total cold run time: 70596 ms
Total hot run time: 54514 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2066288544
<details>
<summary>TPC-DS: <b>Total hot run time: 185014 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f9deb79579005bbac6f4b8999a97e635a5a7d34, data reload: false
query1 904 361 360 360
query2 6924 2540 2233 2233
query3 6650 223 212 212
query4 24446 21440 21350 21350
query5 4190 417 399 399
query6 265 173 175 173
query7 4594 296 282 282
query8 251 193 194 193
query9 8498 2321 2281 2281
query10 601 247 242 242
query11 14725 14276 14260 14260
query12 142 91 87 87
query13 1639 376 358 358
query14 9601 7484 7350 7350
query15 249 188 190 188
query16 8104 288 261 261
query17 1895 594 558 558
query18 2095 295 276 276
query19 195 156 153 153
query20 95 88 95 88
query21 238 126 127 126
query22 5186 4938 4930 4930
query23 34100 33243 33293 33243
query24 12067 2964 2939 2939
query25 666 366 371 366
query26 1774 160 152 152
query27 3135 310 311 310
query28 7685 2001 1985 1985
query29 1019 591 587 587
query30 291 177 174 174
query31 940 704 743 704
query32 95 52 56 52
query33 744 254 241 241
query34 1082 483 467 467
query35 826 688 677 677
query36 1041 907 931 907
query37 277 70 68 68
query38 3351 3238 3218 3218
query39 1627 1525 1514 1514
query40 277 121 130 121
query41 48 44 43 43
query42 108 93 102 93
query43 557 526 539 526
query44 1225 717 723 717
query45 278 268 273 268
query46 1092 713 777 713
query47 1936 1846 1857 1846
query48 365 290 288 288
query49 1198 391 396 391
query50 756 377 388 377
query51 6664 6682 6684 6682
query52 102 96 88 88
query53 360 296 285 285
query54 346 224 229 224
query55 76 72 72 72
query56 239 225 219 219
query57 1248 1137 1127 1127
query58 223 207 193 193
query59 3360 3059 3235 3059
query60 245 225 228 225
query61 89 86 89 86
query62 625 455 448 448
query63 300 286 276 276
query64 6243 4125 3759 3759
query65 3146 3023 3056 3023
query66 1375 349 351 349
query67 15603 15046 15394 15046
query68 5320 539 546 539
query69 547 303 292 292
query70 1184 1189 1182 1182
query71 1416 1274 1264 1264
query72 6427 2599 2485 2485
query73 726 317 322 317
query74 6962 6501 6551 6501
query75 3424 2615 2635 2615
query76 3817 977 1011 977
query77 632 274 267 267
query78 11089 10423 10401 10401
query79 3084 528 526 526
query80 1796 428 425 425
query81 531 238 246 238
query82 878 96 101 96
query83 330 176 168 168
query84 263 91 83 83
query85 1604 287 259 259
query86 485 321 301 301
query87 3490 3319 3328 3319
query88 4306 2331 2348 2331
query89 502 369 361 361
query90 1982 181 178 178
query91 143 97 94 94
query92 64 49 47 47
query93 4780 526 508 508
query94 1237 184 246 184
query95 387 294 296 294
query96 605 273 268 268
query97 3166 2946 2940 2940
query98 235 221 217 217
query99 1250 849 829 829
Total cold run time: 292953 ms
Total hot run time: 185014 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bugfix](hive)Modify the method used to obtain the txnId [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33883:
URL: https://github.com/apache/doris/pull/33883#issuecomment-2065846016
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org