You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Devaraj K (JIRA)" <ji...@apache.org> on 2012/11/24 15:30:58 UTC
[jira] [Created] (YARN-241) Node Manager fails to launch containers
after restart in secure mode
Devaraj K created YARN-241:
------------------------------
Summary: Node Manager fails to launch containers after restart in secure mode
Key: YARN-241
URL: https://issues.apache.org/jira/browse/YARN-241
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.1-alpha, 2.0.2-alpha
Reporter: Devaraj K
Priority: Blocker
After restarting the Node Manager it fails to launch containers with the below exception.
{code:xml}
2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8048: readAndProcess threw exception java.lang.IllegalArgumentException: Invalid key to HMAC computation from client 158.1.131.10. Count of bytes read: 0
java.lang.IllegalArgumentException: Invalid key to HMAC computation
at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
Caused by: java.security.InvalidKeyException: No installed provider supports this key: javax.crypto.spec.SecretKeySpec
at javax.crypto.Mac.a(DashoA13*..)
at javax.crypto.Mac.init(DashoA13*..)
at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
... 11 more
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-241) Node Manager fails to launch containers
after NM restart in secure mode
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K updated YARN-241:
---------------------------
Summary: Node Manager fails to launch containers after NM restart in secure mode (was: Node Manager fails to launch containers after restart in secure mode)
> Node Manager fails to launch containers after NM restart in secure mode
> -----------------------------------------------------------------------
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha, 2.0.1-alpha
> Reporter: Devaraj K
> Priority: Blocker
>
> After restarting the Node Manager it fails to launch containers with the below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8048: readAndProcess threw exception java.lang.IllegalArgumentException: Invalid key to HMAC computation from client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch
containers after NM restart in secure mode
Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503836#comment-13503836 ]
Daryn Sharp commented on YARN-241:
----------------------------------
Is this maybe caused by a race condition where the NM is receiving a container token before the RM registration completes and it receives the secret keys for the container tokens?
> Node Manager fails to launch containers after NM restart in secure mode
> -----------------------------------------------------------------------
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha, 2.0.1-alpha
> Reporter: Devaraj K
> Priority: Blocker
>
> After restarting the Node Manager it fails to launch containers with the below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8048: readAndProcess threw exception java.lang.IllegalArgumentException: Invalid key to HMAC computation from client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-241) Node Manager fails to launch
containers after NM restart in secure mode
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503867#comment-13503867 ]
Devaraj K commented on YARN-241:
--------------------------------
As per my observation when I debug, It is having the secret key and mac.init(key) is failing. It fails for all subsequent invocations. If we try with new mac instance with same secret key it succeeds.
> Node Manager fails to launch containers after NM restart in secure mode
> -----------------------------------------------------------------------
>
> Key: YARN-241
> URL: https://issues.apache.org/jira/browse/YARN-241
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.0.2-alpha, 2.0.1-alpha
> Reporter: Devaraj K
> Priority: Blocker
>
> After restarting the Node Manager it fails to launch containers with the below exception.
> {code:xml}
> 2012-11-24 17:21:56,141 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8048: readAndProcess threw exception java.lang.IllegalArgumentException: Invalid key to HMAC computation from client 158.1.131.10. Count of bytes read: 0
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:153)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:109)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.retrievePassword(ContainerTokenSecretManager.java:44)
> at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.getPassword(SaslRpcServer.java:194)
> at org.apache.hadoop.security.SaslRpcServer$SaslDigestCallbackHandler.handle(SaslRpcServer.java:220)
> at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:568)
> at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:226)
> at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:1199)
> at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:710)
> at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:509)
> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:484)
> Caused by: java.security.InvalidKeyException: No installed provider supports this key: javax.crypto.spec.SecretKeySpec
> at javax.crypto.Mac.a(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:151)
> ... 11 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira