You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org> on 2024/07/10 09:32:37 UTC
[PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
SWJTU-ZhangLei opened a new pull request, #37614:
URL: https://github.com/apache/doris/pull/37614
* https://forums.oracle.com/ords/apexds/post/je-log-checksumexception-2812
* When meeting disk damage or other reason described in the oracle forums and fe cannot start due to `com.sleepycat.je.log.ChecksumException`, we add a param `ignore_bdbje_log_checksum_read` to ignore the exception, but there is no guarantee of correctness for bdbje kv data
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #37614:
URL: https://github.com/apache/doris/pull/37614
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #37614:
URL: https://github.com/apache/doris/pull/37614#issuecomment-2220016815
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37614:
URL: https://github.com/apache/doris/pull/37614#issuecomment-2220086046
<details>
<summary>TPC-DS: <b>Total hot run time: 204067 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c, data reload: false
query1 933 417 380 380
query2 6539 2682 2902 2682
query3 6917 202 197 197
query4 21020 18048 17931 17931
query5 19740 6550 6547 6547
query6 299 220 238 220
query7 4155 298 309 298
query8 438 498 414 414
query9 3048 2649 2561 2561
query10 439 298 309 298
query11 11302 10744 10709 10709
query12 124 73 76 73
query13 5596 699 710 699
query14 18112 13502 13909 13502
query15 365 241 248 241
query16 6460 291 264 264
query17 1700 1449 882 882
query18 2299 417 408 408
query19 220 151 153 151
query20 77 77 78 77
query21 187 95 96 95
query22 5141 4996 4954 4954
query23 32554 32171 31874 31874
query24 6937 6496 6543 6496
query25 518 438 426 426
query26 529 159 160 159
query27 1877 302 299 299
query28 6099 2339 2292 2292
query29 2897 2727 2810 2727
query30 247 166 165 165
query31 915 727 733 727
query32 76 65 59 59
query33 400 256 267 256
query34 829 485 476 476
query35 1133 941 983 941
query36 1291 1354 1352 1352
query37 86 61 60 60
query38 3093 2966 2977 2966
query39 1353 1326 1317 1317
query40 199 98 98 98
query41 53 45 44 44
query42 80 82 78 78
query43 667 737 670 670
query44 1110 715 730 715
query45 250 239 236 236
query46 1228 969 970 969
query47 1856 1690 1954 1690
query48 1030 730 721 721
query49 613 380 377 377
query50 874 585 632 585
query51 4823 4657 4629 4629
query52 93 86 91 86
query53 449 330 334 330
query54 2677 2448 2470 2448
query55 84 80 76 76
query56 246 208 213 208
query57 1126 1107 1145 1107
query58 216 211 198 198
query59 4169 3811 3838 3811
query60 223 218 208 208
query61 100 96 97 96
query62 836 462 459 459
query63 501 345 350 345
query64 2495 1549 1407 1407
query65 3593 3546 3621 3546
query66 788 377 388 377
query67 17950 15804 15223 15223
query68 8935 649 667 649
query69 576 347 342 342
query70 1443 1441 1343 1343
query71 419 310 323 310
query72 6509 3508 3498 3498
query73 728 321 317 317
query74 6315 5975 5879 5879
query75 5325 3686 3670 3670
query76 5409 1160 1241 1160
query77 944 263 257 257
query78 12422 12390 11972 11972
query79 13143 635 653 635
query80 826 405 391 391
query81 494 233 236 233
query82 1174 100 105 100
query83 177 132 138 132
query84 255 69 71 69
query85 791 335 331 331
query86 322 301 290 290
query87 3213 3032 3021 3021
query88 6001 2320 2319 2319
query89 431 307 290 290
query90 2116 211 214 211
query91 176 142 140 140
query92 61 54 52 52
query93 6268 560 611 560
query94 859 210 208 208
query95 1098 1056 1051 1051
query96 655 332 327 327
query97 6505 6344 6443 6344
query98 184 172 173 172
query99 2975 882 1011 882
Total cold run time: 321023 ms
Total hot run time: 204067 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37614:
URL: https://github.com/apache/doris/pull/37614#issuecomment-2220097141
<details>
<summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c, data reload: false
query1 0.02 0.02 0.02
query2 0.07 0.02 0.02
query3 0.25 0.05 0.04
query4 1.80 0.10 0.07
query5 0.52 0.53 0.52
query6 1.26 0.62 0.61
query7 0.01 0.00 0.01
query8 0.04 0.02 0.02
query9 0.51 0.48 0.48
query10 0.55 0.53 0.54
query11 0.12 0.09 0.08
query12 0.12 0.09 0.09
query13 0.62 0.61 0.61
query14 0.78 0.78 0.79
query15 0.78 0.76 0.76
query16 0.37 0.37 0.40
query17 1.01 1.02 1.01
query18 0.21 0.25 0.25
query19 1.88 1.83 1.83
query20 0.01 0.01 0.02
query21 15.47 0.55 0.53
query22 2.11 2.28 1.43
query23 17.38 1.02 0.91
query24 4.49 1.73 1.30
query25 0.37 0.08 0.06
query26 0.63 0.16 0.16
query27 0.04 0.05 0.04
query28 7.64 0.71 0.73
query29 12.62 2.24 2.05
query30 0.55 0.53 0.52
query31 2.81 0.38 0.37
query32 3.38 0.50 0.49
query33 3.07 3.03 3.13
query34 15.24 4.79 4.78
query35 4.85 4.83 4.82
query36 1.04 1.01 1.01
query37 0.06 0.04 0.04
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.14 0.14
query41 0.08 0.01 0.02
query42 0.02 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 103.02 s
Total hot run time: 30.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37614:
URL: https://github.com/apache/doris/pull/37614#issuecomment-2220103160
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37614:
URL: https://github.com/apache/doris/pull/37614#issuecomment-2220060352
<details>
<summary>TPC-H: <b>Total hot run time: 50020 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6398c4fde2a86e616d15fba845a0c8f6f059132c, data reload: false
------ Round 1 ----------------------------------
q1 17831 4422 4364 4364
q2 2033 158 146 146
q3 10270 1891 1948 1891
q4 10146 1299 1326 1299
q5 8458 3928 3919 3919
q6 235 126 126 126
q7 2044 1635 1601 1601
q8 9317 2755 2731 2731
q9 10979 10275 10332 10275
q10 8639 3547 3480 3480
q11 435 248 253 248
q12 471 308 309 308
q13 18349 3950 4033 3950
q14 348 338 338 338
q15 511 463 463 463
q16 680 581 579 579
q17 1143 982 973 973
q18 7309 6878 7023 6878
q19 1791 1651 1640 1640
q20 547 303 300 300
q21 4450 4139 4059 4059
q22 532 454 452 452
Total cold run time: 116518 ms
Total hot run time: 50020 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4349 4318 4316 4316
q2 323 227 220 220
q3 4220 4180 4148 4148
q4 2773 2759 2753 2753
q5 7149 7127 7086 7086
q6 238 122 123 122
q7 3245 2861 2825 2825
q8 4346 4512 4529 4512
q9 16889 16709 16810 16709
q10 4222 4268 4314 4268
q11 752 674 687 674
q12 1023 848 875 848
q13 7331 3774 3776 3774
q14 461 432 420 420
q15 517 461 456 456
q16 745 670 673 670
q17 3795 3886 3846 3846
q18 8832 8717 8760 8717
q19 1722 1734 1681 1681
q20 2403 2148 2141 2141
q21 8477 8546 8537 8537
q22 1062 973 1001 973
Total cold run time: 84874 ms
Total hot run time: 79696 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment (#31247) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37614:
URL: https://github.com/apache/doris/pull/37614#issuecomment-2220016732
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org