You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liutang123 (via GitHub)" <gi...@apache.org> on 2024/03/01 13:46:46 UTC
[PR] [bug](meta) exit when get RollbackException in observer. [doris]
liutang123 opened a new pull request, #31687:
URL: https://github.com/apache/doris/pull/31687
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1978094523
@liutang123 Can you check all RollbackException and InsufficientException and fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #31687:
URL: https://github.com/apache/doris/pull/31687
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "liutang123 (via GitHub)" <gi...@apache.org>.
liutang123 commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1975629108
> @liutang123 #28192 Can you check your version which met this problem has container this pr? the pr 28192 is for solving the problem
I took a look at #28192 and I am not sure can it resolve this case.
`isReplicaTxnAreMatched` will check the first log in a database, but the last log may not be committed to the majority.
As follows:
Old master has a databases: 1000 (1000, 1001, 1002,1003, 1004, 1005)
The old master try to commit log 1005, but only commit it to the observer.
The new master will create a database 1005(1005, ... ) and the log 1005 in this database is `OP_MASTER_INFO_CHANGE `.
In observer, log 1000 in database 1000 has been commited in majority but log 1005 may just persisted in local dist and may be read and replay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1973234719
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1977022785
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3a2ab284706cbe5a82cc02e79c974be1b20138b9 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 18.3 seconds inserted 10000000 Rows, about 546K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1982313190
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1975659553
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1976971121
<details>
<summary>TPC-H: <b>Total hot run time: 37653 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3a2ab284706cbe5a82cc02e79c974be1b20138b9, data reload: false
------ Round 1 ----------------------------------
q1 17654 4062 4086 4062
q2 2012 146 147 146
q3 10548 929 931 929
q4 4666 929 916 916
q5 7596 2870 2984 2870
q6 177 126 124 124
q7 1286 808 815 808
q8 9495 2025 2074 2025
q9 7293 6427 6377 6377
q10 8196 2525 2551 2525
q11 415 223 218 218
q12 754 316 312 312
q13 17954 2916 2929 2916
q14 276 255 248 248
q15 482 443 450 443
q16 487 386 395 386
q17 941 846 805 805
q18 6658 5809 5827 5809
q19 1559 1520 1523 1520
q20 554 274 278 274
q21 7522 3669 3649 3649
q22 786 291 295 291
Total cold run time: 107311 ms
Total hot run time: 37653 ms
----- Round 2, with runtime_filter_mode=off -----
q1 3992 3985 3980 3980
q2 323 232 242 232
q3 2949 2918 2911 2911
q4 1831 1775 1801 1775
q5 5205 5227 5226 5226
q6 220 117 119 117
q7 2276 1845 1814 1814
q8 3207 3248 3268 3248
q9 8509 8510 8464 8464
q10 6145 3725 3690 3690
q11 538 434 448 434
q12 692 559 536 536
q13 11368 2754 2795 2754
q14 271 253 259 253
q15 467 445 431 431
q16 448 424 407 407
q17 1683 1651 1636 1636
q18 7803 7478 7273 7273
q19 4207 1604 1590 1590
q20 1940 1752 1700 1700
q21 4977 4778 4744 4744
q22 507 470 471 470
Total cold run time: 69558 ms
Total hot run time: 53685 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1977005707
<details>
<summary>TPC-DS: <b>Total hot run time: 177936 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3a2ab284706cbe5a82cc02e79c974be1b20138b9, data reload: false
query1 928 354 349 349
query2 7361 2060 2131 2060
query3 6707 214 212 212
query4 27225 20850 21002 20850
query5 4280 3915 534 534
query6 270 178 182 178
query7 4628 311 304 304
query8 249 174 169 169
query9 8482 2197 2176 2176
query10 446 239 244 239
query11 14976 14265 14430 14265
query12 134 92 89 89
query13 1689 438 458 438
query14 9352 7107 7038 7038
query15 230 196 193 193
query16 7234 274 282 274
query17 964 602 578 578
query18 1927 293 288 288
query19 221 169 174 169
query20 94 95 92 92
query21 223 137 128 128
query22 4666 4540 4420 4420
query23 31296 31089 30716 30716
query24 10958 3204 3068 3068
query25 662 390 401 390
query26 1611 170 172 170
query27 2929 355 367 355
query28 6622 1834 1821 1821
query29 1054 635 616 616
query30 295 154 157 154
query31 945 751 751 751
query32 106 65 62 62
query33 779 268 262 262
query34 944 471 505 471
query35 958 803 805 803
query36 939 825 844 825
query37 148 69 69 69
query38 3258 3130 3072 3072
query39 1415 1355 1369 1355
query40 221 119 127 119
query41 60 54 52 52
query42 113 103 105 103
query43 440 415 412 412
query44 1090 726 717 717
query45 212 200 199 199
query46 1050 825 786 786
query47 1636 1542 1550 1542
query48 428 351 362 351
query49 1142 346 354 346
query50 788 391 389 389
query51 6670 6592 6525 6525
query52 114 98 99 98
query53 384 287 292 287
query54 281 249 256 249
query55 89 88 86 86
query56 248 233 228 228
query57 1080 996 1004 996
query58 264 229 236 229
query59 2451 2330 2259 2259
query60 281 292 254 254
query61 118 117 115 115
query62 648 431 412 412
query63 317 283 291 283
query64 5230 3210 3144 3144
query65 3048 3008 3013 3008
query66 1106 320 332 320
query67 15064 14582 14432 14432
query68 13576 562 588 562
query69 724 388 392 388
query70 1838 1076 1127 1076
query71 596 287 284 284
query72 10260 2600 2500 2500
query73 3105 340 340 340
query74 7095 6778 6934 6778
query75 5972 2692 2628 2628
query76 7184 1094 1226 1094
query77 731 256 256 256
query78 10352 9654 9548 9548
query79 7289 527 529 527
query80 932 428 422 422
query81 487 221 217 217
query82 245 95 97 95
query83 217 155 149 149
query84 283 80 80 80
query85 1082 340 328 328
query86 373 297 296 296
query87 3432 3193 3158 3158
query88 2833 2379 2373 2373
query89 516 370 371 370
query90 2062 189 175 175
query91 162 131 131 131
query92 61 53 52 52
query93 4115 526 512 512
query94 1328 190 186 186
query95 462 356 362 356
query96 599 263 267 263
query97 4035 3867 3888 3867
query98 245 218 216 216
query99 1062 786 765 765
Total cold run time: 305031 ms
Total hot run time: 177936 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "liutang123 (via GitHub)" <gi...@apache.org>.
liutang123 commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1973294971
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1977016513
<details>
<summary>ClickBench: <b>Total hot run time: 30.68 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3a2ab284706cbe5a82cc02e79c974be1b20138b9, data reload: false
query1 0.04 0.04 0.03
query2 0.06 0.03 0.03
query3 0.24 0.05 0.06
query4 1.66 0.10 0.10
query5 0.50 0.50 0.50
query6 1.20 0.66 0.66
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.56 0.54 0.50
query10 0.56 0.54 0.56
query11 0.13 0.10 0.10
query12 0.13 0.10 0.11
query13 0.57 0.57 0.57
query14 0.74 0.74 0.77
query15 0.83 0.79 0.80
query16 0.37 0.38 0.38
query17 1.01 0.97 0.97
query18 0.27 0.27 0.25
query19 1.82 1.70 1.74
query20 0.02 0.01 0.02
query21 15.41 0.62 0.62
query22 2.81 3.97 2.03
query23 17.16 1.10 1.03
query24 2.12 0.40 0.58
query25 0.30 0.04 0.04
query26 0.17 0.14 0.14
query27 0.03 0.03 0.04
query28 12.16 0.83 0.85
query29 12.73 3.43 3.37
query30 0.61 0.56 0.55
query31 2.81 0.36 0.34
query32 3.36 0.43 0.44
query33 2.92 2.89 2.87
query34 15.48 4.30 4.33
query35 4.33 4.33 4.32
query36 1.08 1.03 1.02
query37 0.07 0.05 0.05
query38 0.05 0.03 0.03
query39 0.03 0.02 0.02
query40 0.17 0.14 0.14
query41 0.07 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.03 0.02
Total cold run time: 104.7 s
Total hot run time: 30.68 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1975488130
@liutang123 https://github.com/apache/doris/pull/28192
Can you check your version which met this problem has container this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](meta) exit when get RollbackException in observer. [doris]
Posted by "liutang123 (via GitHub)" <gi...@apache.org>.
liutang123 commented on PR #31687:
URL: https://github.com/apache/doris/pull/31687#issuecomment-1976881262
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org