You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org> on 2024/02/22 02:09:35 UTC
[PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
SWJTU-ZhangLei opened a new pull request, #31247:
URL: https://github.com/apache/doris/pull/31247
* https://forums.oracle.com/ords/apexds/post/je-log-checksumexception-2812
* When meeting disk damage or other reason described in the oracle forums and fe cannot start due to `com.sleepycat.je.log.ChecksumException`, we add a param `ignore_bdbje_log_checksum_read` to ignore the exception, but there is no guarantee of correctness for bdbje kv data
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958776679
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 44d7b0f13296f629161f188177cefaa3745b3c5b with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #31247:
URL: https://github.com/apache/doris/pull/31247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958531788
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-2067873666
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958768964
<details>
<summary>TPC-DS: <b>Total hot run time: 176411 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44d7b0f13296f629161f188177cefaa3745b3c5b, data reload: false
query1 938 361 348 348
query2 6512 1738 1764 1738
query3 6701 204 200 200
query4 23261 21254 21186 21186
query5 4336 370 370 370
query6 253 174 168 168
query7 4705 313 288 288
query8 256 208 199 199
query9 8465 2779 2769 2769
query10 416 239 226 226
query11 15200 14523 14484 14484
query12 140 82 83 82
query13 1677 418 417 417
query14 9660 7704 7646 7646
query15 221 192 192 192
query16 7613 254 240 240
query17 1412 549 522 522
query18 1942 266 264 264
query19 188 149 152 149
query20 85 80 83 80
query21 187 112 116 112
query22 4837 4754 4707 4707
query23 32430 31398 31480 31398
query24 12699 3449 3383 3383
query25 638 370 358 358
query26 1890 157 164 157
query27 3049 311 316 311
query28 6658 1846 1841 1841
query29 1117 612 610 610
query30 287 133 145 133
query31 944 758 756 756
query32 94 59 57 57
query33 720 228 226 226
query34 1076 483 501 483
query35 955 830 861 830
query36 1039 955 877 877
query37 175 63 60 60
query38 3308 3207 3168 3168
query39 1378 1311 1327 1311
query40 283 108 104 104
query41 37 37 36 36
query42 106 99 99 99
query43 477 442 436 436
query44 1065 674 687 674
query45 194 186 178 178
query46 1042 807 768 768
query47 1637 1504 1559 1504
query48 423 335 348 335
query49 1205 297 300 297
query50 766 370 374 370
query51 4487 4315 4291 4291
query52 108 95 92 92
query53 402 304 302 302
query54 289 227 223 223
query55 92 83 80 80
query56 217 196 199 196
query57 1055 968 918 918
query58 220 196 190 190
query59 2300 2166 2201 2166
query60 249 218 214 214
query61 83 82 83 82
query62 602 370 378 370
query63 324 271 277 271
query64 6344 3042 3106 3042
query65 3310 3251 3239 3239
query66 1335 323 321 321
query67 14514 14353 14361 14353
query68 5090 549 545 545
query69 508 359 354 354
query70 1256 1135 1173 1135
query71 416 256 258 256
query72 6346 2750 2594 2594
query73 707 309 310 309
query74 6971 6500 6428 6428
query75 3188 2588 2540 2540
query76 3307 1083 1203 1083
query77 354 235 231 231
query78 9510 8893 8797 8797
query79 984 500 489 489
query80 514 358 352 352
query81 434 201 202 201
query82 161 87 83 83
query83 141 122 123 122
query84 222 76 76 76
query85 993 338 339 338
query86 293 306 311 306
query87 3435 3319 3266 3266
query88 2735 2275 2276 2275
query89 443 364 357 357
query90 1833 159 160 159
query91 152 128 126 126
query92 58 49 54 49
query93 1000 507 493 493
query94 1114 174 181 174
query95 432 335 335 335
query96 563 257 261 257
query97 4462 4316 4271 4271
query98 218 198 198 198
query99 1053 723 715 715
Total cold run time: 269521 ms
Total hot run time: 176411 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958758138
<details>
<summary>TPC-H: <b>Total hot run time: 40852 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44d7b0f13296f629161f188177cefaa3745b3c5b, data reload: false
------ Round 1 ----------------------------------
q1 17632 4991 4879 4879
q2 2044 141 137 137
q3 10586 1019 973 973
q4 4727 941 962 941
q5 7627 3091 3172 3091
q6 195 132 127 127
q7 1238 776 778 776
q8 9306 2063 2045 2045
q9 7432 6564 6549 6549
q10 8317 2622 2619 2619
q11 414 202 199 199
q12 758 326 326 326
q13 17930 3600 3583 3583
q14 292 256 256 256
q15 615 504 531 504
q16 484 401 424 401
q17 921 862 838 838
q18 7375 6619 6631 6619
q19 1554 1486 1489 1486
q20 547 273 264 264
q21 6676 3925 3923 3923
q22 862 333 316 316
Total cold run time: 107532 ms
Total hot run time: 40852 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4889 4882 4856 4856
q2 289 186 187 186
q3 3612 3573 3575 3573
q4 2533 2518 2532 2518
q5 5734 5733 5753 5733
q6 209 124 126 124
q7 2243 1655 1624 1624
q8 3050 3141 3093 3093
q9 8705 8717 8726 8717
q10 6817 4217 4219 4217
q11 509 364 363 363
q12 763 537 529 529
q13 4314 3388 3384 3384
q14 252 244 232 232
q15 629 503 533 503
q16 473 429 427 427
q17 1697 1609 1624 1609
q18 8327 7790 7606 7606
q19 1623 1631 1620 1620
q20 2096 1850 1816 1816
q21 6541 6151 6210 6151
q22 602 486 497 486
Total cold run time: 65907 ms
Total hot run time: 59367 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-2067880164
<details>
<summary>TPC-H: <b>Total hot run time: 38488 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b99da858669b3df45d067ee1288efccae449aac9, data reload: false
------ Round 1 ----------------------------------
q1 17590 4570 4419 4419
q2 2007 178 182 178
q3 10457 1115 1138 1115
q4 10181 814 734 734
q5 7508 2679 2637 2637
q6 215 130 133 130
q7 1009 619 584 584
q8 9218 2063 2042 2042
q9 7453 6583 6545 6545
q10 8485 3511 3542 3511
q11 452 232 228 228
q12 459 222 222 222
q13 17769 2901 2917 2901
q14 285 225 222 222
q15 516 476 482 476
q16 513 389 379 379
q17 956 673 743 673
q18 7410 6677 6622 6622
q19 6791 1555 1504 1504
q20 682 318 317 317
q21 3530 2750 2812 2750
q22 360 312 299 299
Total cold run time: 113846 ms
Total hot run time: 38488 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4349 4247 4217 4217
q2 392 261 275 261
q3 2988 2745 2709 2709
q4 1877 1610 1585 1585
q5 5369 5295 5295 5295
q6 211 124 125 124
q7 2224 1890 1862 1862
q8 3231 3327 3325 3325
q9 8641 8547 8584 8547
q10 3910 3705 3661 3661
q11 561 478 497 478
q12 769 607 605 605
q13 17345 2948 2961 2948
q14 312 278 272 272
q15 519 472 468 468
q16 473 419 427 419
q17 1781 1499 1472 1472
q18 8098 7689 8058 7689
q19 1690 1607 1559 1559
q20 2031 1827 1843 1827
q21 8329 5017 4935 4935
q22 563 469 457 457
Total cold run time: 75663 ms
Total hot run time: 54715 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-2069441019
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958531881
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1976490281
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958773825
<details>
<summary>ClickBench: <b>Total hot run time: 30.48 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44d7b0f13296f629161f188177cefaa3745b3c5b, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.22 0.07 0.07
query4 1.67 0.09 0.09
query5 0.49 0.50 0.49
query6 1.33 0.62 0.64
query7 0.01 0.01 0.02
query8 0.04 0.03 0.03
query9 0.53 0.47 0.45
query10 0.50 0.49 0.49
query11 0.13 0.09 0.10
query12 0.11 0.10 0.10
query13 0.59 0.60 0.58
query14 0.79 0.76 0.79
query15 0.82 0.81 0.79
query16 0.34 0.33 0.34
query17 0.90 0.92 0.94
query18 0.17 0.17 0.18
query19 1.80 1.70 1.69
query20 0.01 0.02 0.01
query21 15.40 0.60 0.59
query22 2.91 3.80 1.82
query23 17.54 0.96 0.97
query24 2.10 0.55 0.38
query25 0.61 0.06 0.06
query26 0.15 0.14 0.14
query27 0.05 0.05 0.05
query28 11.94 0.82 0.83
query29 12.56 3.36 3.34
query30 0.53 0.48 0.48
query31 2.78 0.36 0.36
query32 3.37 0.47 0.47
query33 3.13 3.11 3.19
query34 15.35 4.52 4.48
query35 4.51 4.48 4.52
query36 1.06 0.94 0.95
query37 0.07 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.01 0.02
query40 0.17 0.14 0.15
query41 0.07 0.02 0.01
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 104.94 s
Total hot run time: 30.48 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #31247:
URL: https://github.com/apache/doris/pull/31247#discussion_r1498531014
##########
fe/fe-common/src/main/java/org/apache/doris/common/Config.java:
##########
@@ -2488,6 +2488,9 @@ public class Config extends ConfigBase {
options = {"default", "ranger-doris"})
public static String access_controller_type = "default";
+ // https://forums.oracle.com/ords/apexds/post/je-log-checksumexception-2812
+ @ConfField
+ public static boolean ignore_bdbje_log_checksum_read = false;
Review Comment:
Add a comment here to describe when to set it to true, and what's the defects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1958728430
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](fe) Add `ignore_bdbje_log_checksum_read` for BDBEnvironment [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31247:
URL: https://github.com/apache/doris/pull/31247#issuecomment-1976490221
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org