You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "CalvinKirs (via GitHub)" <gi...@apache.org> on 2025/05/26 07:53:50 UTC
[PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
CalvinKirs opened a new pull request, #51245:
URL: https://github.com/apache/doris/pull/51245
### What problem does this PR solve?
When using HadoopCatalog with Kerberos authentication, write operations fail during doCommit because the underlying FileSystem is accessed without proper credentials.
This patch ensures that the Kerberos credentials are available during commit-time FS operations, allowing data writes to succeed under kerberos environments.
`Caused by: org.apache.iceberg.exceptions.RuntimeIOException: Failed to refresh the table at org.apache.iceberg.hadoop.HadoopTableOperations.refresh(HadoopTableOperations.java:128) ~[iceberg-core-1.6.1.jar:?] at org.apache.iceberg.Transactions.newTransaction(Transactions.java:63) ~[iceberg-core-1.6.1.jar:?] at org.apache.iceberg.BaseTable.newTransaction(BaseTable.java:240) ~[iceberg-core-1.6.1.jar:?] at org.apache.doris.datasource.iceberg.IcebergTransaction.beginInsert(IcebergTransaction.java:79) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.IcebergInsertExecutor.beforeExec(IcebergInsertExecutor.java:55) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.AbstractInsertExecutor.executeSingleInsert(AbstractInsertExecutor.java:198) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.runInternal(InsertIntoTableCommand.java:472) ~[doris-f
e.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.run(InsertIntoTableCommand.java:166) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:771) ~[doris-fe.jar:1.2-SNAPSHOT] ... 17 moreCaused by: java.io.IOException: DestHost:destPort hadoop-master:8520 , LocalHost:localPort vm-204/172.20.57.204:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?] at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?] at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?] at java.lang.reflect.Con
structor.newInstance(Constructor.java:480) ~[?:?] at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:930) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:905) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1571) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1513) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1410) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139) ~[hadoop-common-3.3.6.jar:?] at jdk.proxy3.$Proxy162.getFileInfo(Unknown Source) ~[?:?] at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:966) ~[hadoop-hdfs-client-3.3.6.jar:?] at jd
k.internal.reflect.GeneratedMethodAccessor53.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:433) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362) ~[hadoop-common-3.3.6.jar:?] at jdk.proxy3.$Proxy163.getFileInfo(Unknown Source) ~[?:?] at org.apache.hadoop.hdfs.DFSCli
ent.getFileInfo(DFSClient.java:1739) ~[hadoop-hdfs-client-3.3.6.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1829) ~[hadoop-hdfs-client-3.3.6.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1826) ~[hadoop-hdfs-client-3.3.6.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.3.6.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1841) ~[hadoop-hdfs-client-3.3.6.jar:?] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1862) ~[hadoop-common-3.3.6.jar:?] at org.apache.iceberg.hadoop.HadoopTableOperations.getMetadataFile(HadoopTableOperations.java:243) ~[iceberg-core-1.6.1.jar:?] at org.apache.iceberg.hadoop.HadoopTableOperations.refresh(HadoopTableOperations.java:108) ~[iceberg-core-1.6.1.jar:?] at org.apache.iceberg.Transactions.newTransaction(Transactions.java:
63) ~[iceberg-core-1.6.1.jar:?] at org.apache.iceberg.BaseTable.newTransaction(BaseTable.java:240) ~[iceberg-core-1.6.1.jar:?] at org.apache.doris.datasource.iceberg.IcebergTransaction.beginInsert(IcebergTransaction.java:79) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.IcebergInsertExecutor.beforeExec(IcebergInsertExecutor.java:55) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.AbstractInsertExecutor.executeSingleInsert(AbstractInsertExecutor.java:198) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.runInternal(InsertIntoTableCommand.java:472) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand.run(InsertIntoTableCommand.java:166) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:771) ~[doris-fe.jar:1.2-SNAPSHOT] ... 17
more `
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2908896481
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2909093860
<details>
<summary>TPC-DS: <b>Total hot run time: 186148 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e95fa14a4e972a8258327f937a8de690cd58cda, data reload: false
query1 1065 474 500 474
query2 6565 1809 1807 1807
query3 6743 229 222 222
query4 26648 23320 23398 23320
query5 4342 634 477 477
query6 313 209 208 208
query7 4624 500 293 293
query8 297 261 238 238
query9 8634 2601 2593 2593
query10 477 343 265 265
query11 15954 15015 14785 14785
query12 160 108 106 106
query13 1643 532 394 394
query14 9293 6132 6258 6132
query15 202 204 169 169
query16 7226 634 475 475
query17 1165 714 552 552
query18 1958 391 289 289
query19 190 191 156 156
query20 117 119 121 119
query21 224 124 111 111
query22 4111 4197 3983 3983
query23 34063 33191 32974 32974
query24 8494 2392 2417 2392
query25 576 495 421 421
query26 1230 267 160 160
query27 2754 519 349 349
query28 4338 2116 2099 2099
query29 806 594 453 453
query30 283 215 185 185
query31 941 863 788 788
query32 77 68 65 65
query33 569 368 327 327
query34 807 883 521 521
query35 792 799 736 736
query36 959 1013 901 901
query37 112 99 86 86
query38 4235 4114 4066 4066
query39 1496 1405 1435 1405
query40 218 125 113 113
query41 65 104 57 57
query42 129 122 118 118
query43 506 481 470 470
query44 1307 830 826 826
query45 176 175 164 164
query46 842 1025 643 643
query47 1810 1842 1794 1794
query48 399 428 312 312
query49 770 532 426 426
query50 660 681 407 407
query51 4140 4170 4135 4135
query52 116 111 108 108
query53 227 257 193 193
query54 594 570 506 506
query55 86 87 87 87
query56 317 301 288 288
query57 1144 1179 1126 1126
query58 270 260 267 260
query59 2648 2684 2569 2569
query60 377 334 308 308
query61 124 126 125 125
query62 792 745 659 659
query63 221 194 195 194
query64 4377 1033 654 654
query65 4343 4279 4273 4273
query66 1152 417 309 309
query67 15937 16128 15566 15566
query68 8694 899 521 521
query69 471 311 320 311
query70 1165 1150 1097 1097
query71 478 321 305 305
query72 5362 4777 4850 4777
query73 731 654 382 382
query74 9228 8892 8691 8691
query75 4130 3196 2662 2662
query76 3677 1185 740 740
query77 808 386 303 303
query78 10079 10385 9339 9339
query79 1837 784 573 573
query80 593 532 452 452
query81 469 261 221 221
query82 438 127 98 98
query83 250 252 228 228
query84 246 114 91 91
query85 789 358 370 358
query86 336 315 314 314
query87 4492 4407 4293 4293
query88 3404 2266 2292 2266
query89 387 315 289 289
query90 1956 209 206 206
query91 143 140 114 114
query92 74 60 63 60
query93 1267 954 583 583
query94 687 418 320 320
query95 384 293 284 284
query96 504 570 280 280
query97 2755 2769 2696 2696
query98 238 206 201 201
query99 1340 1426 1307 1307
Total cold run time: 275642 ms
Total hot run time: 186148 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2909765893
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #51245:
URL: https://github.com/apache/doris/pull/51245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "CalvinKirs (via GitHub)" <gi...@apache.org>.
CalvinKirs commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2908902501
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2909059236
<details>
<summary>TPC-H: <b>Total hot run time: 33980 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e95fa14a4e972a8258327f937a8de690cd58cda, data reload: false
------ Round 1 ----------------------------------
q1 26260 5123 5037 5037
q2 2113 279 193 193
q3 10495 1302 705 705
q4 10226 993 531 531
q5 7671 2412 2340 2340
q6 181 160 139 139
q7 940 736 622 622
q8 9314 1310 1059 1059
q9 6866 5087 5093 5087
q10 6840 2311 1919 1919
q11 535 289 293 289
q12 352 352 215 215
q13 17769 3706 3125 3125
q14 235 228 213 213
q15 544 473 490 473
q16 432 442 382 382
q17 606 864 365 365
q18 7793 7365 7168 7168
q19 1579 974 559 559
q20 345 356 223 223
q21 3941 3206 2373 2373
q22 1018 1010 963 963
Total cold run time: 116055 ms
Total hot run time: 33980 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5147 5137 5093 5093
q2 240 335 232 232
q3 2145 2709 2338 2338
q4 1351 1825 1520 1520
q5 4604 4414 4352 4352
q6 213 166 127 127
q7 1993 1918 1769 1769
q8 2569 2598 2542 2542
q9 7198 7189 7130 7130
q10 2992 3212 2742 2742
q11 573 512 514 512
q12 683 790 613 613
q13 3489 3920 3333 3333
q14 283 314 277 277
q15 513 469 468 468
q16 427 517 449 449
q17 1142 1559 1370 1370
q18 7640 7587 7440 7440
q19 819 785 834 785
q20 1990 1979 1831 1831
q21 4706 4314 4327 4314
q22 1069 1034 978 978
Total cold run time: 51786 ms
Total hot run time: 50215 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2909111407
<details>
<summary>ClickBench: <b>Total hot run time: 29.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e95fa14a4e972a8258327f937a8de690cd58cda, data reload: false
query1 0.04 0.05 0.04
query2 0.14 0.11 0.12
query3 0.26 0.20 0.20
query4 1.59 0.20 0.11
query5 0.44 0.40 0.42
query6 1.14 0.67 0.66
query7 0.02 0.02 0.02
query8 0.04 0.03 0.04
query9 0.59 0.53 0.52
query10 0.58 0.58 0.58
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.62 0.60 0.59
query14 0.79 0.82 0.81
query15 0.88 0.86 0.85
query16 0.39 0.40 0.39
query17 1.08 1.06 1.03
query18 0.22 0.21 0.21
query19 1.92 1.81 1.90
query20 0.01 0.01 0.02
query21 15.41 0.94 0.57
query22 0.75 1.17 0.77
query23 14.83 1.40 0.67
query24 6.86 2.06 0.88
query25 0.50 0.27 0.07
query26 0.64 0.16 0.13
query27 0.05 0.06 0.04
query28 9.51 0.91 0.45
query29 12.54 3.97 3.30
query30 0.25 0.09 0.07
query31 2.82 0.60 0.38
query32 3.23 0.56 0.47
query33 2.98 3.12 3.00
query34 16.00 5.14 4.45
query35 4.50 4.52 4.48
query36 0.66 0.49 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.02
query40 0.16 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.03
query43 0.03 0.03 0.03
Total cold run time: 103.06 s
Total hot run time: 29.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](Iceberg-hadoop-catalog)Fix Kerberos-authenticated HadoopCatalog insert failures due to missing kerberos credentials [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #51245:
URL: https://github.com/apache/doris/pull/51245#issuecomment-2909766227
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org