You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ashish Kumar (Jira)" <ji...@apache.org> on 2024/01/08 11:23:00 UTC
[jira] [Resolved] (HDDS-9191) OM initialization fails when securely talking to SCM via Kerberos Tickets
[ https://issues.apache.org/jira/browse/HDDS-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashish Kumar resolved HDDS-9191.
--------------------------------
Resolution: Cannot Reproduce
Closing this now, If happen please reopen it.
> OM initialization fails when securely talking to SCM via Kerberos Tickets
> --------------------------------------------------------------------------
>
> Key: HDDS-9191
> URL: https://issues.apache.org/jira/browse/HDDS-9191
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM, SCM
> Affects Versions: 1.3.0, 1.4.0
> Environment: OS: Red Hat 8.8
> Java: 11
> OM server: ddl07zone01
> SCM server: ddl07zone02
> ozone-site.xml security settings:
> {code:java}
> <!-- SCM Security Settings -->
> <property>
> <name>hdds.scm.kerberos.principal</name>
> <value>scm/_HOST@ROOT.LOCAL</value>
> </property><property>
> <name>hdds.scm.kerberos.keytab.file</name>
> <value>/opt/ozone/ozone.keytab</value>
> </property><!-- OM Security Settings -->
> <property>
> <name>ozone.om.kerberos.principal</name>
> <value>om/_HOST@ROOT.LOCAL</value>
> </property><property>
> <name>ozone.om.kerberos.keytab.file</name>
> <value>/opt/ozone/ozone.keytab</value>
> </property> {code}
> Reporter: David Ayres
> Assignee: Ashish Kumar
> Priority: Major
>
> When enabling cluster security via kerberos and providing UPN and keytab file in the ozone-site.xml initialization of OM fails with the following error:
> {code:java}
> 2023-08-18 11:27:50,208 [main] DEBUG security.UserGroupInformation: PrivilegedAction [as: om/ddl07zone01.root.local@ROOT.LOCAL (auth:KERBEROS)][action: org.apache.hadoop.ipc.Client$Connection$1@42714a7]
> java.lang.Exception
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
> at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:752)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:856)
> at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1677)
> at org.apache.hadoop.ipc.Client.call(Client.java:1502)
> at org.apache.hadoop.ipc.Client.call(Client.java:1455)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:235)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:122)
> at com.sun.proxy.$Proxy31.send(Unknown Source)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy31.send(Unknown Source)
> at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.submitRequest(ScmBlockLocationProtocolClientSideTranslatorPB.java:121)
> at org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolClientSideTranslatorPB.getScmInfo(ScmBlockLocationProtocolClientSideTranslatorPB.java:266)
> at org.apache.hadoop.hdds.utils.HAUtils.getScmInfo(HAUtils.java:100)
> at org.apache.hadoop.ozone.om.OzoneManager.omInit(OzoneManager.java:1233)
> at org.apache.hadoop.ozone.om.OzoneManagerStarter$OMStarterHelper.init(OzoneManagerStarter.java:204)
> at org.apache.hadoop.ozone.om.OzoneManagerStarter.initOm(OzoneManagerStarter.java:102)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at picocli.CommandLine.executeUserObject(CommandLine.java:1972)
> at picocli.CommandLine.access$1300(CommandLine.java:145)
> at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
> at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
> at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
> at picocli.CommandLine.execute(CommandLine.java:2078)
> at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:100)
> at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:91)
> at org.apache.hadoop.ozone.om.OzoneManagerStarter.main(OzoneManagerStarter.java:58)
> 2023-08-18 11:27:50,208 [main] DEBUG ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed {code}
> On the SCM side I get this error:
> {code:java}
> 2023-08-17 16:05:53,628 [Socket Reader #1 for port 9863] WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.236.152.241:35587:null (GSS initiate failed) with true cause: (GSS initiate failed)2023-08-17 16:05:53,628 [Socket Reader #1 for port 9863] DEBUG org.apache.hadoop.ipc.Server: Socket Reader #1 for port 9863: processOneRpc from client 10.236.152.241:35587 threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)]]
> {code}
> One thing that stands out:
> {code:java}
> 2023-08-18 11:27:51,438 [main] DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdds.scm.protocolPB.ScmBlockLocationProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal="", serverPrincipal="hdds.scm.kerberos.principal") {code}
> {{Is clientPrincipal supposed to be blank?}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org