You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karam Singh (Created) (JIRA)" <ji...@apache.org> on 2011/12/12 06:42:31 UTC
[jira] [Created] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
--------------------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-3531
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager, scheduler
Affects Versions: 0.23.1
Reporter: Karam Singh
Filling this Jira a bit late
Started 350 cluster
sbummited large sleep job.
Foud that job was not running as RM has not allocated resouces to it.
{code}
2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
type NODE_UPDATE to the scheduler
java.lang.IllegalArgumentException: Invalid key to HMAC computation
at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.security.InvalidKeyException: Secret key expected
at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
at javax.crypto.Mac.init(DashoA13*..)
at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
... 14 more
{code}
As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170291#comment-13170291 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #928 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/928/])
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169934#comment-13169934 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1508 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1508/])
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169604#comment-13169604 ]
Hadoop QA commented on MAPREDUCE-3531:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507389/MR-3531.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1449//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1449//console
This message is automatically generated.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169638#comment-13169638 ]
Siddharth Seth commented on MAPREDUCE-3531:
-------------------------------------------
The change looks good to me. I'd prefer removing the test case if it can't reproduce the problem each time.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169936#comment-13169936 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #281 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/281/])
merge MAPREDUCE-3531 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169927#comment-13169927 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1435 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1435/])
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169721#comment-13169721 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3531:
----------------------------------------------------
+! for the fix as is. Atleast we could use that brute-force test to reproduce it sometimes, validating the fix is mostly the correct one.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169717#comment-13169717 ]
Hadoop QA commented on MAPREDUCE-3531:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507428/MR-3531.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1452//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1452//console
This message is automatically generated.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3531:
-------------------------------------
Priority: Critical (was: Major)
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Priority: Critical
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3531:
-------------------------------------------
Attachment: MR-3531.txt
Adding in a patch to fix the issue (Just changed HashMap to ConcurrentHashMap).
I also added in a test, but I don't really like the test that much. It tries to brute force reproduce the problem. On my box it takes about 1.8 sec to run so it should not be too big of an issue right now.
However, when trying to run without the fix I have seen it reproduce the problem, but I have also seen it essentially get into a living deadlock where all of the cores are pegged at 100% utilization, and sometimes only a kill -9 stops it form running. I am fine with removing the test, because I don't think it adds much value, but I am also fine with leaving it in.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3531:
-------------------------------------------
Attachment: MR-3531.txt
This patch does not have the test in it.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3531:
--------------------------------------
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Status: Open (was: Patch Available)
Bobby, could you please upload another patch without the unit test.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3531:
-------------------------------------------
Fix Version/s: 0.24.0
0.23.1
Target Version/s: 0.23.1, 0.24.0
Status: Patch Available (was: Open)
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169937#comment-13169937 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1459 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1459/])
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Contributed by Robert Joseph Evans)
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214593
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3531:
-------------------------------------------
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Status: Patch Available (was: Open)
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3531:
--------------------------------------
Resolution: Fixed
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk and branch-0.23. Thanks Bobby.
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169941#comment-13169941 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #303 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/303/])
merge MAPREDUCE-3531 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3531:
-----------------------------------------------
Priority: Blocker (was: Critical)
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170195#comment-13170195 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #128 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/128/])
merge MAPREDUCE-3531 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170225#comment-13170225 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #108 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/108/])
merge MAPREDUCE-3531 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169939#comment-13169939 ]
Hudson commented on MAPREDUCE-3531:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #291 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/291/])
merge MAPREDUCE-3531 from trunk
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214595
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/ContainerTokenSecretManager.java
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Blocker
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MR-3531.txt, MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3531) Sometimes
java.lang.IllegalArgumentException: Invalid key to HMAC computation in
NODE_UPDATE also causing RM to stop scheduling
Posted by "Robert Joseph Evans (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans reassigned MAPREDUCE-3531:
----------------------------------------------
Assignee: Robert Joseph Evans
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3531
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Assignee: Robert Joseph Evans
> Priority: Critical
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
> at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
> atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
> at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
> at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
> at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
> at javax.crypto.Mac.init(DashoA13*..)
> at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
> ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira