You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Chris Riccomini (JIRA)" <ji...@apache.org> on 2012/09/21 20:05:07 UTC
[jira] [Created] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Chris Riccomini created MAPREDUCE-4672:
------------------------------------------
Summary: RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
Key: MAPREDUCE-4672
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: resourcemanager
Affects Versions: 0.23.1
Reporter: Chris Riccomini
Hey Guys,
I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all nodes (to simulate a complete failure), the containers timeout and fail, as expected.
What's unexpected is that my log then starts filling with:
2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
Thanks!
Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462057#comment-13462057 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-4672:
----------------------------------------------------
Okie, then can you check if you are getting a reboot as part of AMResponse in the allocate calls? You should act on that if you are writing your custom AM. MR AM already does something like that.
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462036#comment-13462036 ]
Chris Riccomini commented on MAPREDUCE-4672:
--------------------------------------------
Hey Vinod,
In this case, I've killed ALL NMs, but the RMs are still running.
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462062#comment-13462062 ]
Chris Riccomini commented on MAPREDUCE-4672:
--------------------------------------------
Hey Vinod,
There is no AM running. All NMs are down.
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Riccomini updated MAPREDUCE-4672:
---------------------------------------
Description:
Hey Guys,
I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs (to simulate a complete failure), the containers timeout and fail, as expected.
What's unexpected is that my log then starts filling with:
2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
Thanks!
Chris
was:
Hey Guys,
I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all nodes (to simulate a complete failure), the containers timeout and fail, as expected.
What's unexpected is that my log then starts filling with:
2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
Thanks!
Chris
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs (to simulate a complete failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli reassigned MAPREDUCE-4672:
--------------------------------------------------
Assignee: Vinod Kumar Vavilapalli
In this case, you just killed the NMs and the AM is still running? In that case, RM realizes that the node running AM went down, assumes that AM should also go down and marks it as an invalid App.
It does ask the AM to shut-down as part of the ping-response but it looks like your AM isn't handling that correctly.
{code}
AllocateResponse allocateResponse = recordFactory
.newRecordInstance(AllocateResponse.class);
AMResponse lastResponse = responseMap.get(appAttemptId);
if (lastResponse == null) {
LOG.error("AppAttemptId doesnt exist in cache " + appAttemptId);
allocateResponse.setAMResponse(reboot);
return allocateResponse;
}
{code}
See {{AMResponse.getReboot()}}
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462813#comment-13462813 ]
Chris Riccomini commented on MAPREDUCE-4672:
--------------------------------------------
Ahhh, I see. So I should use the mapred AM as an example of how to gracefully shut down in the case where the AM is running with no NM? The AMResponse will denote that I should shut down:
getReboot()
Should the ApplicationMaster reboot for being horribly out-of-sync with the ResourceManager as deigned by getResponseId()?
Thanks, I'll use this.
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462185#comment-13462185 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-4672:
----------------------------------------------------
Chris, yes I understand that your NM process is down. But when NM goes down, it doesn't kill its containers as of now. So I am sure your AM container process is still running (from the call trace ApplicationMasterService.java:allocate(247) ).
You have to kill this AM process either manually or by handling the value of AMResponse.getReboot() in code.
If this isn't your current job, it should be some stale AM from before.
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
> Assignee: Vinod Kumar Vavilapalli
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4672) RM with lost NMs results in
massive log of AppAttemptId doesnt exist in cache
Posted by "Chris Riccomini (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Riccomini updated MAPREDUCE-4672:
---------------------------------------
Description:
Hey Guys,
I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
What's unexpected is that my log then starts filling with:
2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
Thanks!
Chris
was:
Hey Guys,
I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs (to simulate a complete failure), the containers timeout and fail, as expected.
What's unexpected is that my log then starts filling with:
2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
Thanks!
Chris
> RM with lost NMs results in massive log of AppAttemptId doesnt exist in cache
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-4672
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4672
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 0.23.1
> Reporter: Chris Riccomini
>
> Hey Guys,
> I'm running a 9 node cluster with 8 NMs and a single RM node. If I run an app master and have that app master start a container, then shut down all NMs, but leave the RM up (to simulate a failure), the containers timeout and fail, as expected.
> What's unexpected is that my log then starts filling with:
> 2012-09-21 18:02:02,614 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:03,617 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:04,618 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:05,620 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:06,621 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:07,623 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> 2012-09-21 18:02:08,624 ERROR resourcemanager.ApplicationMasterService (ApplicationMasterService.java:allocate(247)) - AppAttemptId doesnt exist in cache appattempt_1348248013002_0001_000001
> Is there any way to shut this off/fix it? It just keeps going forever, until I bounce the RM node.
> Thanks!
> Chris
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira