You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/19 02:36:51 UTC
[PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Gabriel39 opened a new pull request, #69247:
URL: https://github.com/apache/doris/pull/69247
### What problem does this PR solve?
Problem Summary:
Queries against Kerberos-protected Paimon storage can fail with `Failed to find latest snapshot id` and `Client cannot authenticate via: [TOKEN, KERBEROS]`. Resolving a Table under authentication does not cover snapshot I/O performed after the resolver returns. In addition, storage configured with `hdfs.authentication.*` can initialize without a plugin-side Kerberos authenticator because the selection checks only the raw legacy Hadoop key.
Authenticate actual snapshot-cache loads, partition-cache key loads, and the complete time-travel resolution. Reuse the metadata-read wrapper and select the storage authenticator from the effective Hadoop configuration, including normalized storage properties. HMS and storage identities remain separate; cache behavior and empty-table snapshot semantics are preserved.
### Release note
Fix Kerberos-authenticated Paimon snapshot reads and recognize normalized HDFS authentication settings when selecting the plugin storage identity.
### Check List (For Author)
- [x] Unit Test
- [x] Regression test
- [x] Manual test
- Behavior changed: authenticated Paimon metadata reads retain the storage identity through deferred file I/O.
- Documentation needed: No; no configuration key or public API added.
Validation:
- Reproduced 11 snapshot authentication scenarios before the fix. Added a negative authentication/cache recovery control; all 12 pass after the fix.
- Reproduced the normalized-storage-authenticator assertion failure before fixing the legacy-only gate.
- Paimon module `install` with `Paimon*` tests: 823 tests, zero failures/errors, one skipped live-connectivity test. Checkstyle and plugin packaging passed.
- Local KDC/HMS/HDFS using the original failing DEBUG/ASAN runtime: both Paimon cases failed at the first snapshot read before the patch. Replacing the Paimon plugin with the built artifact made both local suites pass (two suites, zero failed/fatal/skipped), including all four Kerberos HMS property variants, JNI/native reads, incremental reads, time travel, system tables, and the added REFRESH assertion.
- Local adaptations replace endpoints/keytab paths and exclude unrelated object-store variants. The migration fixture has matching columns and four rows, created as local Paimon tables; it does not reproduce the original Hive-migrated physical files.
- Ten concurrent FE plans across two Kerberos identities succeeded; a mismatched keytab was rejected. A separate intermittent BE credential error observed with a warm backend is not claimed fixed here; the final SQL run used a fresh backend.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6262643579
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6271310488
<details>
<summary>ClickBench: <b>Total hot run time: 24.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e8c0525bdac77d296cc63d3e4eaf5d00c047999, data reload: false
query1 0.00 0.01 0.00
query2 0.09 0.04 0.04
query3 0.26 0.09 0.09
query4 1.61 0.10 0.10
query5 0.23 0.24 0.25
query6 1.17 0.68 0.67
query7 0.04 0.01 0.03
query8 0.05 0.03 0.03
query9 0.39 0.35 0.35
query10 0.56 0.60 0.57
query11 0.20 0.14 0.15
query12 0.17 0.15 0.14
query13 0.48 0.50 0.49
query14 1.03 1.02 1.02
query15 0.62 0.60 0.60
query16 0.34 0.34 0.36
query17 1.10 1.13 1.12
query18 0.20 0.19 0.18
query19 2.09 2.00 1.97
query20 0.02 0.01 0.01
query21 15.48 0.19 0.14
query22 4.86 0.06 0.05
query23 16.30 0.33 0.12
query24 3.04 0.57 0.50
query25 0.10 0.07 0.07
query26 1.37 0.21 0.16
query27 0.06 0.05 0.05
query28 3.38 0.69 0.36
query29 12.49 4.18 3.33
query30 0.28 0.14 0.10
query31 2.77 0.56 0.32
query32 3.22 0.60 0.50
query33 3.22 3.24 3.16
query34 15.28 3.98 3.37
query35 3.34 3.38 3.31
query36 0.56 0.46 0.42
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.18 0.15 0.14
query41 0.08 0.03 0.03
query42 0.03 0.02 0.03
query43 0.04 0.04 0.04
Total cold run time: 96.91 s
Total hot run time: 24.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6258458737
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6264388230
<details>
<summary>ClickBench: <b>Total hot run time: 24.19 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 86cd8732c93d2f1040dd27ab08d44e4b26fcd82f, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.10 0.09
query5 0.26 0.25 0.24
query6 1.17 0.68 0.67
query7 0.03 0.01 0.01
query8 0.06 0.04 0.04
query9 0.39 0.35 0.36
query10 0.57 0.60 0.58
query11 0.20 0.14 0.14
query12 0.18 0.14 0.14
query13 0.48 0.49 0.49
query14 1.02 1.01 1.01
query15 0.62 0.60 0.60
query16 0.35 0.36 0.34
query17 1.08 1.09 1.15
query18 0.20 0.18 0.18
query19 2.07 1.95 2.02
query20 0.02 0.01 0.01
query21 15.46 0.19 0.13
query22 4.93 0.05 0.05
query23 16.29 0.34 0.12
query24 2.81 0.54 0.33
query25 0.13 0.05 0.07
query26 1.12 0.21 0.16
query27 0.05 0.04 0.03
query28 3.02 0.80 0.37
query29 12.47 4.18 3.32
query30 0.28 0.13 0.10
query31 2.78 0.55 0.32
query32 3.22 0.58 0.49
query33 3.30 3.23 3.17
query34 15.40 3.97 3.38
query35 3.38 3.34 3.33
query36 0.56 0.44 0.40
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.02 0.02
query40 0.18 0.15 0.15
query41 0.09 0.04 0.03
query42 0.03 0.03 0.03
query43 0.05 0.03 0.03
Total cold run time: 96.4 s
Total hot run time: 24.19 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6258454916
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6268617171
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6266296972
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6259152752
<details>
<summary>TPC-H: <b>Total hot run time: 29596 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b60a00f3a64da16cfa6ab443753c9d62d42b304, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17607 5623 5525 5525
q2 2119 328 279 279
q3 10210 1413 838 838
q4 4675 486 342 342
q5 7441 800 528 528
q6 188 182 155 155
q7 770 804 602 602
q8 9237 1353 1157 1157
q9 5637 4277 4282 4277
q10 6809 1305 1049 1049
q11 434 255 237 237
q12 639 402 281 281
q13 18069 3517 2988 2988
q14 280 275 244 244
q15 q16 728 730 674 674
q17 1191 762 584 584
q18 7018 6242 6183 6183
q19 1105 956 608 608
q20 394 358 223 223
q21 5493 2532 2601 2532
q22 402 394 290 290
Total cold run time: 100446 ms
Total hot run time: 29596 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6566 6552 6423 6423
q2 775 599 585 585
q3 5592 5720 5348 5348
q4 2100 2137 1575 1575
q5 5910 5781 5709 5709
q6 241 183 145 145
q7 2250 2037 1832 1832
q8 3225 2594 2556 2556
q9 7648 7601 7644 7601
q10 3664 3617 3231 3231
q11 604 408 387 387
q12 682 693 488 488
q13 3299 3637 2943 2943
q14 283 288 258 258
q15 q16 669 686 616 616
q17 7833 7012 6850 6850
q18 13208 12278 13150 12278
q19 927 788 785 785
q20 2197 2190 1936 1936
q21 5872 4732 4827 4732
q22 495 456 408 408
Total cold run time: 74040 ms
Total hot run time: 66686 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6267974945
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69247_86cd8732c93d2f1040dd27ab08d44e4b26fcd82f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69247_86cd8732c93d2f1040dd27ab08d44e4b26fcd82f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6271225052
<details>
<summary>TPC-DS: <b>Total hot run time: 150745 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e8c0525bdac77d296cc63d3e4eaf5d00c047999, data reload: false
query5 4304 578 440 440
query6 439 206 198 198
query7 4802 444 232 232
query8 337 161 150 150
query9 8775 3132 3117 3117
query10 492 308 252 252
query11 6065 3865 3567 3567
query12 146 91 87 87
query13 1241 427 308 308
query14 6119 3884 3632 3632
query14_1 3428 3413 3415 3413
query15 207 194 185 185
query16 953 421 406 406
query17 880 666 549 549
query18 2420 431 311 311
query19 198 167 136 136
query20 84 84 82 82
query21 215 129 106 106
query22 12981 12995 12743 12743
query23 13625 12832 12510 12510
query23_1 12598 12432 12620 12432
query24 7370 798 454 454
query24_1 463 456 463 456
query25 503 395 328 328
query26 1263 251 136 136
query27 2776 437 263 263
query28 4678 1943 1975 1943
query29 1548 566 437 437
query30 305 214 181 181
query31 1002 936 821 821
query32 137 87 87 87
query33 504 287 227 227
query34 881 852 491 491
query35 759 763 641 641
query36 808 809 733 733
query37 124 102 72 72
query38 1793 1738 1675 1675
query39 781 797 742 742
query39_1 732 726 734 726
query40 215 116 91 91
query41 66 67 68 67
query42 95 84 80 80
query43 341 362 301 301
query44 1306 683 698 683
query45 175 170 165 165
query46 835 967 552 552
query47 2463 2914 2777 2777
query48 293 302 216 216
query49 575 395 283 283
query50 689 270 206 206
query51 10255 10534 10370 10370
query52 75 85 72 72
query53 181 205 151 151
query54 236 192 180 180
query55 69 69 63 63
query56 247 230 197 197
query57 1667 1586 1544 1544
query58 282 257 247 247
query59 2094 2103 1883 1883
query60 284 250 225 225
query61 149 167 168 167
query62 394 339 309 309
query63 193 164 159 159
query64 2890 1080 912 912
query65 3344 3281 3267 3267
query66 1800 431 306 306
query67 20315 20271 20047 20047
query68 3017 1035 636 636
query69 412 315 264 264
query70 889 822 788 788
query71 316 245 283 245
query72 2775 2600 2315 2315
query73 536 530 299 299
query74 4676 4521 4392 4392
query75 2371 2334 1936 1936
query76 3101 1028 603 603
query77 351 364 262 262
query78 9721 9762 9125 9125
query79 989 884 511 511
query80 757 418 341 341
query81 556 320 266 266
query82 570 131 101 101
query83 286 196 185 185
query84 303 129 99 99
query85 942 417 370 370
query86 346 266 229 229
query87 2009 1964 1796 1796
query88 3673 2672 2637 2637
query89 347 279 246 246
query90 1719 161 167 161
query91 160 154 127 127
query92 95 91 89 89
query93 1022 1000 573 573
query94 504 298 266 266
query95 563 323 324 323
query96 669 472 232 232
query97 2456 2444 2316 2316
query98 164 153 156 153
query99 768 766 655 655
Total cold run time: 235172 ms
Total hot run time: 150745 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #69247:
URL: https://github.com/apache/doris/pull/69247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6260765672
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6260765990
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6264131360
<details>
<summary>TPC-H: <b>Total hot run time: 28766 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 86cd8732c93d2f1040dd27ab08d44e4b26fcd82f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17584 5150 5061 5061
q2 2099 321 267 267
q3 10231 1409 826 826
q4 4675 469 337 337
q5 7441 821 526 526
q6 182 188 154 154
q7 769 789 590 590
q8 9246 1296 1100 1100
q9 5558 4213 4403 4213
q10 6843 1288 1044 1044
q11 433 243 220 220
q12 642 403 288 288
q13 18090 3633 3038 3038
q14 283 269 246 246
q15 q16 727 721 663 663
q17 1223 748 587 587
q18 7068 6194 6145 6145
q19 1235 986 613 613
q20 398 341 228 228
q21 6082 2341 2566 2341
q22 386 331 279 279
Total cold run time: 101195 ms
Total hot run time: 28766 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6009 5902 6175 5902
q2 753 585 541 541
q3 5576 5880 5420 5420
q4 2031 2146 1592 1592
q5 5770 5850 5568 5568
q6 247 203 146 146
q7 2204 2015 1849 1849
q8 2949 2575 2592 2575
q9 7945 7880 7537 7537
q10 3644 3568 3181 3181
q11 574 412 380 380
q12 666 680 510 510
q13 3233 3608 3011 3011
q14 285 286 274 274
q15 q16 662 702 608 608
q17 7802 7025 6914 6914
q18 12963 12192 12900 12192
q19 911 804 805 804
q20 2124 2159 1903 1903
q21 5634 4580 4686 4580
q22 501 460 400 400
Total cold run time: 72483 ms
Total hot run time: 65887 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6260151579
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6261448198
<details>
<summary>ClickBench: <b>Total hot run time: 24.66 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c5d8569a085a4e33c248f3bc2aacf4ea307fadf5, data reload: false
query1 0.01 0.00 0.00
query2 0.11 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.12 0.12
query5 0.27 0.24 0.25
query6 1.17 0.68 0.68
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.40 0.35 0.34
query10 0.59 0.59 0.58
query11 0.21 0.14 0.14
query12 0.19 0.15 0.15
query13 0.50 0.50 0.50
query14 1.05 1.03 1.02
query15 0.65 0.61 0.62
query16 0.35 0.36 0.33
query17 1.16 1.09 1.11
query18 0.23 0.21 0.21
query19 2.06 1.96 1.97
query20 0.02 0.01 0.02
query21 15.51 0.28 0.14
query22 4.41 0.05 0.05
query23 16.25 0.32 0.12
query24 3.20 0.55 0.42
query25 0.12 0.06 0.05
query26 1.04 0.25 0.16
query27 0.05 0.04 0.03
query28 1.79 0.64 0.41
query29 12.53 4.20 3.31
query30 0.30 0.16 0.12
query31 2.78 0.57 0.34
query32 3.23 0.61 0.51
query33 3.20 3.26 3.24
query34 15.50 3.99 3.40
query35 3.34 3.35 3.36
query36 0.56 0.46 0.47
query37 0.10 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.04 0.03
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 95.27 s
Total hot run time: 24.66 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6262643132
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6263073119
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6268616653
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69247:
URL: https://github.com/apache/doris/pull/69247#discussion_r4309946247
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java:
##########
@@ -216,9 +216,11 @@ private HadoopAuthenticator pluginAuthenticator() {
*/
static HadoopAuthenticator buildPluginAuthenticator(Map<String, String> properties,
Map<String, String> storageHadoopConfig) {
- if ("kerberos".equalsIgnoreCase(properties.get(HADOOP_SECURITY_AUTHENTICATION))) {
- return HadoopAuthenticator.getHadoopAuthenticator(
- PaimonCatalogFactory.buildHadoopConfiguration(properties, storageHadoopConfig));
+ // Storage bindings normalize hdfs.authentication.* and XML settings into Hadoop keys.
+ // Gating on the raw legacy key would leave FileIO using the plugin's unauthenticated UGI.
+ Configuration conf = PaimonCatalogFactory.buildHadoopConfiguration(properties, storageHadoopConfig);
+ if ("kerberos".equalsIgnoreCase(conf.get(HADOOP_SECURITY_AUTHENTICATION))) {
+ return HadoopAuthenticator.getHadoopAuthenticator(conf);
Review Comment:
[P2] Preserve inherited UGI when Kerberos credentials are absent
If FE's `core-site.xml` supplies `hadoop.security.authentication=kerberos`, but the catalog has no Doris principal/keytab pair (e.g. it relies on an existing Kerberos ticket-cache login), this new gate calls `getHadoopAuthenticator(conf)`, which falls back to `HadoopSimpleAuthenticator` and defaults to user `hadoop`. `TcclPinningConnectorContext` treats that non-null result as authoritative, so FileIO initializes its [HDFS client](https://raw.githubusercontent.com/apache/hadoop/rel/release-3.4.2/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) as a credential-free SIMPLE subject instead of the prior current/login UGI. Paimon's secured wrapper is added after client construction and cannot replace that captured RPC identity. A cold catalog access then fails against secured HDFS; the same identity change can cause permission failures on simple HDFS. Keep the previous null/delegate behavior when the resolved config is the SIMPLE fallback, whi
le retaining Kerberos selection for normalized configurations with credentials.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6262649943
Fixed the failing `test_paimon_write_variant_errors` suite in `483c3884928`.
`try_parse_to_variant('not-json')` returns SQL NULL before the Paimon sink. The stale expected output assumed a Variant string. Updated the explanation and regenerated the expected output without changing the input or removing assertions.
Locally ran the complete original suite using the exact `c5d8569a085` CI binary, Spark 4/Paimon 1.3.1, and isolated object storage. Reproduced the same row 21 mismatch, then regenerated the output and reran the complete suite successfully: 1 suite, 0 failures, 0 fatal errors, 0 skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6264303113
<details>
<summary>TPC-DS: <b>Total hot run time: 150631 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 86cd8732c93d2f1040dd27ab08d44e4b26fcd82f, data reload: false
query5 4316 584 432 432
query6 425 198 187 187
query7 4806 434 222 222
query8 318 161 155 155
query9 8792 3083 3063 3063
query10 473 318 274 274
query11 5958 3857 3563 3563
query12 150 99 85 85
query13 1253 459 340 340
query14 6093 3918 3623 3623
query14_1 3433 3444 3426 3426
query15 212 208 178 178
query16 982 441 428 428
query17 864 687 557 557
query18 2417 436 319 319
query19 209 168 140 140
query20 86 85 85 85
query21 215 127 114 114
query22 12962 12965 12834 12834
query23 13509 12945 12577 12577
query23_1 12746 12433 12492 12433
query24 7385 856 461 461
query24_1 463 457 479 457
query25 514 388 335 335
query26 1249 251 133 133
query27 2795 433 268 268
query28 4657 1941 1927 1927
query29 1521 589 440 440
query30 301 223 183 183
query31 996 927 809 809
query32 139 89 86 86
query33 502 293 235 235
query34 889 836 494 494
query35 731 762 648 648
query36 807 815 716 716
query37 115 94 79 79
query38 1756 1728 1673 1673
query39 803 799 747 747
query39_1 725 731 728 728
query40 217 116 94 94
query41 69 61 62 61
query42 84 83 84 83
query43 345 360 317 317
query44 1308 677 680 677
query45 176 179 163 163
query46 830 959 560 560
query47 2846 3002 2694 2694
query48 298 307 215 215
query49 568 394 290 290
query50 687 277 206 206
query51 10266 10435 10492 10435
query52 76 86 81 81
query53 185 202 148 148
query54 230 195 187 187
query55 69 64 63 63
query56 251 226 203 203
query57 1677 1606 1556 1556
query58 283 245 252 245
query59 2174 2109 1994 1994
query60 296 244 259 244
query61 147 150 145 145
query62 391 326 302 302
query63 191 154 156 154
query64 2734 895 783 783
query65 3300 3245 3285 3245
query66 1756 411 291 291
query67 20654 20136 20029 20029
query68 3165 991 599 599
query69 383 282 243 243
query70 852 773 761 761
query71 289 234 206 206
query72 2657 2544 2297 2297
query73 508 522 295 295
query74 4621 4490 4337 4337
query75 2356 2288 1903 1903
query76 2325 1030 593 593
query77 346 376 266 266
query78 9665 9774 9170 9170
query79 883 840 511 511
query80 469 427 348 348
query81 510 319 273 273
query82 204 133 103 103
query83 193 192 172 172
query84 285 118 96 96
query85 794 414 377 377
query86 254 239 230 230
query87 2012 1918 1825 1825
query88 3613 2675 2623 2623
query89 313 276 247 247
query90 2119 165 164 164
query91 159 152 124 124
query92 95 87 84 84
query93 990 980 582 582
query94 450 308 257 257
query95 580 326 372 326
query96 670 482 233 233
query97 2457 2428 2320 2320
query98 164 159 146 146
query99 760 776 641 641
Total cold run time: 233581 ms
Total hot run time: 150631 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6261379483
<details>
<summary>TPC-DS: <b>Total hot run time: 152636 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5d8569a085a4e33c248f3bc2aacf4ea307fadf5, data reload: false
query5 4309 587 447 447
query6 433 217 194 194
query7 4797 451 227 227
query8 337 174 160 160
query9 8760 4030 4068 4030
query10 498 318 261 261
query11 6094 3800 3538 3538
query12 144 94 89 89
query13 1252 458 329 329
query14 6149 4073 3818 3818
query14_1 3556 3582 3558 3558
query15 210 201 184 184
query16 943 452 425 425
query17 864 683 543 543
query18 2437 438 314 314
query19 205 169 142 142
query20 86 86 82 82
query21 209 139 113 113
query22 13032 13123 12759 12759
query23 13378 12755 12465 12465
query23_1 12518 12514 12487 12487
query24 7133 835 466 466
query24_1 483 479 483 479
query25 543 422 376 376
query26 1293 268 142 142
query27 2765 435 278 278
query28 4652 1952 1959 1952
query29 1576 608 469 469
query30 308 225 188 188
query31 1024 921 815 815
query32 142 93 94 93
query33 533 302 254 254
query34 967 857 497 497
query35 752 785 650 650
query36 842 807 732 732
query37 129 105 100 100
query38 1793 1768 1729 1729
query39 791 811 754 754
query39_1 742 722 750 722
query40 220 122 100 100
query41 73 70 68 68
query42 92 88 82 82
query43 366 379 339 339
query44 1291 692 698 692
query45 185 190 168 168
query46 862 967 572 572
query47 2869 3017 2979 2979
query48 294 304 210 210
query49 571 401 290 290
query50 702 277 219 219
query51 10628 10447 10342 10342
query52 85 87 71 71
query53 184 205 154 154
query54 230 189 187 187
query55 79 69 64 64
query56 226 235 229 229
query57 1609 1582 1568 1568
query58 300 269 264 264
query59 2126 2194 1957 1957
query60 292 234 227 227
query61 150 148 150 148
query62 395 340 285 285
query63 199 165 164 164
query64 2719 953 807 807
query65 3317 3253 3265 3253
query66 1770 420 310 310
query67 20374 20187 19982 19982
query68 3002 991 611 611
query69 376 283 247 247
query70 849 790 811 790
query71 287 228 233 228
query72 2690 2581 2286 2286
query73 521 525 298 298
query74 4631 4471 4332 4332
query75 2349 2303 1965 1965
query76 2306 1026 624 624
query77 336 362 285 285
query78 9722 9793 9159 9159
query79 982 920 524 524
query80 533 429 375 375
query81 530 317 269 269
query82 590 146 111 111
query83 305 203 184 184
query84 312 123 103 103
query85 859 425 397 397
query86 334 239 236 236
query87 1995 1957 1816 1816
query88 3679 2726 2698 2698
query89 338 295 261 261
query90 1890 188 183 183
query91 171 154 132 132
query92 102 94 82 82
query93 1029 1023 572 572
query94 468 312 287 287
query95 599 401 322 322
query96 671 531 231 231
query97 2463 2446 2323 2323
query98 167 157 146 146
query99 785 764 645 645
Total cold run time: 234592 ms
Total hot run time: 152636 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6263083582
Rebase validation (86cd8732c93):
- Resolved actual merge conflicts against master while retaining the new upstream Variant write, UPDATE, and MERGE assertions. Original PR fixes are unchanged; range-diff confirms they were preserved.
- Paimon module `install`: 832 tests, zero failures/errors, one external-connectivity skip; connector Checkstyle and packaging passed.
- The complete local regression reproductions and passing reruns above used the original failing CI artifacts before this rebase. The newly expanded upstream suite has not been rerun locally on a rebuilt master runtime; full integration validation on the rebased head is delegated to the newly requested CI run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6274070967
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69247_6e8c0525bdac77d296cc63d3e4eaf5d00c047999_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69247_6e8c0525bdac77d296cc63d3e4eaf5d00c047999_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6258454331
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6260151264
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6266297952
CHAR-padding regression follow-up:
- The upstream MAP-key guard (#69101) rejects STRUCT keys before the Paimon sink, while the CHAR-padding suite (#69083) still expected those writes to succeed or reach writer collision checks.
- All four complex-key groups now assert the specific analysis error for both former success/collision inputs, and verify their tables remain empty. Primitive CHAR-key padding/collision checks and partition, ORC, bucket-routing, rowkind, UPDATE/DELETE, and MERGE coverage are retained. Direct writer normalization tests remain unchanged.
- Reproduced the original full-suite failure on the CI runtime built from `86cd8732c93`. Regenerated expected output with the regression runner, then independently ran the complete `test_paimon_write_char_padding` and `test_paimon_write_variant_errors` suites: two passed, zero failures/fatal errors/skips. Only local endpoints were adapted; product binaries were unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6271051535
<details>
<summary>TPC-H: <b>Total hot run time: 28917 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e8c0525bdac77d296cc63d3e4eaf5d00c047999, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17588 5158 5048 5048
q2 2090 304 252 252
q3 10253 1417 802 802
q4 4682 472 345 345
q5 7469 805 539 539
q6 179 181 150 150
q7 751 795 606 606
q8 9254 1350 1102 1102
q9 5507 4212 4174 4174
q10 6829 1292 1053 1053
q11 428 243 226 226
q12 641 404 288 288
q13 18090 3673 3037 3037
q14 272 267 250 250
q15 q16 717 712 684 684
q17 1224 755 579 579
q18 7036 6164 6143 6143
q19 1123 969 617 617
q20 387 340 223 223
q21 5862 2526 2559 2526
q22 370 334 273 273
Total cold run time: 100752 ms
Total hot run time: 28917 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5960 5986 6074 5986
q2 691 586 579 579
q3 5572 5761 5292 5292
q4 2041 2162 1571 1571
q5 5830 5686 5752 5686
q6 244 183 151 151
q7 2207 1954 1770 1770
q8 3039 2651 2612 2612
q9 7974 7830 7718 7718
q10 3640 3571 3114 3114
q11 573 399 373 373
q12 668 674 504 504
q13 3287 3598 3052 3052
q14 281 283 257 257
q15 q16 677 683 615 615
q17 7726 7070 6855 6855
q18 12961 12220 12933 12220
q19 928 812 800 800
q20 2148 2156 1913 1913
q21 5610 4766 4808 4766
q22 495 479 400 400
Total cold run time: 72552 ms
Total hot run time: 66234 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6261235504
<details>
<summary>TPC-H: <b>Total hot run time: 29357 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5d8569a085a4e33c248f3bc2aacf4ea307fadf5, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17586 5622 5577 5577
q2 2097 314 294 294
q3 10197 1389 844 844
q4 4679 475 352 352
q5 7452 801 535 535
q6 189 192 158 158
q7 774 802 605 605
q8 9251 1355 1054 1054
q9 5499 4289 4253 4253
q10 6828 1291 1014 1014
q11 432 251 231 231
q12 630 404 284 284
q13 18079 3500 2979 2979
q14 275 278 259 259
q15 q16 732 729 672 672
q17 1193 747 569 569
q18 6988 6219 6199 6199
q19 1100 945 608 608
q20 389 333 224 224
q21 5437 2376 2461 2376
q22 359 314 270 270
Total cold run time: 100166 ms
Total hot run time: 29357 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6177 6153 6153 6153
q2 664 569 535 535
q3 5228 5371 4948 4948
q4 2057 2138 1681 1681
q5 5315 5405 5787 5405
q6 269 194 167 167
q7 2279 2098 1878 1878
q8 3175 2746 2872 2746
q9 8169 8110 8147 8110
q10 3791 3706 3362 3362
q11 645 456 433 433
q12 722 720 521 521
q13 3564 4016 3212 3212
q14 325 312 292 292
q15 q16 682 723 641 641
q17 8490 7037 6901 6901
q18 13198 12278 13073 12278
q19 913 785 791 785
q20 2179 2164 1925 1925
q21 6044 4980 5015 4980
q22 490 453 409 409
Total cold run time: 74376 ms
Total hot run time: 67362 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6260769520
Rebased onto master `1eb427f19eb`; the new head is `c5d8569a085`.
The conflict was in `test_paimon_write_thread_lifecycle.groovy`: merged PR #69230 already replaced global JVM/process thread sampling with writer-owned thread checks. Dropped the obsolete sampling-only commit `bce156d8e70` and kept the upstream test unchanged. `git range-diff` confirms that the Paimon authentication patch is unchanged by the rebase.
Local validation after the rebase:
- Paimon module `install` with `Paimon*`: 823 tests, zero failures/errors, one skipped live-connectivity test.
- Checkstyle and plugin packaging passed; `git diff --check` passed.
- The end-to-end Kerberos suites were not rerun for this rebase; their earlier results and limitations remain documented in the PR description.
Updated the description to remove the superseded lifecycle sampling changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6259613981
# FE UT Coverage Report
Increment line coverage `` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69247_1b60a00f3a64da16cfa6ab443753c9d62d42b304/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69247_1b60a00f3a64da16cfa6ab443753c9d62d42b304/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6263072543
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6260115131
<details>
<summary>TPC-H: <b>Total hot run time: 29770 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b60a00f3a64da16cfa6ab443753c9d62d42b304, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17576 5750 5648 5648
q2 2077 300 246 246
q3 10281 1431 838 838
q4 4681 496 346 346
q5 7431 821 545 545
q6 190 194 160 160
q7 789 812 623 623
q8 9281 1369 1146 1146
q9 5613 4280 4290 4280
q10 6840 1332 1027 1027
q11 436 261 239 239
q12 634 433 283 283
q13 18063 3618 3045 3045
q14 286 280 255 255
q15 q16 741 735 695 695
q17 1199 749 579 579
q18 7076 6251 6200 6200
q19 1110 956 593 593
q20 394 340 232 232
q21 4942 2508 2569 2508
q22 448 366 282 282
Total cold run time: 100088 ms
Total hot run time: 29770 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6535 6602 6577 6577
q2 758 656 633 633
q3 5560 5787 5253 5253
q4 2318 2357 1734 1734
q5 5735 5823 5619 5619
q6 238 194 149 149
q7 2358 2076 1892 1892
q8 3161 2692 2655 2655
q9 7583 7541 7521 7521
q10 3775 3655 3324 3324
q11 620 444 423 423
q12 682 700 532 532
q13 3328 3609 3077 3077
q14 290 309 272 272
q15 q16 688 703 639 639
q17 8047 7177 7024 7024
q18 13280 12307 13177 12307
q19 912 786 783 783
q20 2213 2187 1978 1978
q21 6172 5106 5175 5106
q22 525 466 417 417
Total cold run time: 74778 ms
Total hot run time: 67915 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6268618715
Rebase validation (master `41a58afba2e`):
- Resolved the conflict by retaining master's Variant fix from #69261 and removing this PR's duplicate Variant change. The output is identical; only the explanatory comment uses the upstream wording.
- Original product fixes and the CHAR-padding regression fix are preserved. Range-diff confirms the unchanged patches; the Iceberg product patch is also unchanged where applicable.
- Paimon connector `install`: 832 tests, zero failures/errors, one live-connectivity skip; Checkstyle and plugin packaging passed.
- The end-to-end results listed below were obtained before this rebase. Full integration validation of the new head is pending CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](paimon) Authenticate deferred metadata reads and normalized HDFS storage [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69247:
URL: https://github.com/apache/doris/pull/69247#issuecomment-6266296441
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org