You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karam Singh (Commented) (JIRA)" <ji...@apache.org> on 2012/01/11 16:50:40 UTC
[jira] [Commented] (MAPREDUCE-3656) Sort job on 350 scale is
consistently failing with latest MRV2 code
[ https://issues.apache.org/jira/browse/MAPREDUCE-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184148#comment-13184148 ]
Karam Singh commented on MAPREDUCE-3656:
----------------------------------------
For org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_UPDATE at ASSIGNED
Follwing is NM logs messages :
{code}
2012-01-11 13:50:21,402 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=<jobuser> IP=<ip_addres_of_RM> OPERATION=Stop Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1326289061888_0002 CONTAINERID=container_1326289061888_0002_01_000001
2012-01-11 13:50:21,406 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Processing container_1326289061888_0002_01_000001 of type KILL_CONTAINER
2012-01-11 13:50:21,407 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1326289061888_0002_01_000001 transitioned from RUNNING to KILLING
2012-01-11 13:50:21,407 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1326289061888_0002_01_000001
2012-01-11 13:50:21,432 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Processing container_1326289061888_0002_01_000001 of type UPDATE_DIAGNOSTICS_MSG
2012-01-11 13:50:21,449 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Processing container_1326289061888_0002_01_000001 of type CONTAINER_KILLED_ON_REQUEST
{code}
RM logs says :
{code}
012-01-11 13:47:34,326 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: Processing container_1326289061888_0002_01_000001 of type LAUNCHED
2012-01-11 13:47:34,326 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1326289061888_0002_01_000001 Container Transitioned from ACQUIRED to RUNNING
2012-01-11 13:50:22,377 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: Processing container_1326289061888_0002_01_000001 of type KILL
2012-01-11 13:50:22,377 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1326289061888_0002_01_000001 Container Transitioned from RUNNING to KILLED
2012-01-11 13:50:22,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApp: Completed container: container_1326289061888_0002_01_000001 in state: KILLED event:KILL
{code}
Stack of tracke of InvalidStateTransitonException :
{code}
012-01-11 13:48:45,526 ERROR [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle this event at current state for attempt_1326289061888_0002_m_006598_0
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_UPDATE at ASSIGNED
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:919)
at org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:130)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:871)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:863)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:82)
at java.lang.Thread.run(Thread.java:619)
{code}
> Sort job on 350 scale is consistently failing with latest MRV2 code
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-3656
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3656
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, mrv2, resourcemanager
> Affects Versions: 0.23.1
> Reporter: Karam Singh
> Priority: Critical
> Fix For: 0.23.1
>
>
> With the code checked out on last two days.
> Sort Job on 350 node scale with 16800 maps and 680 reduces consistently failing for around last 6 runs
> When around 50% of maps are completed, suddenly job jumps to failed state.
> On looking at NM log, found RM sent Stop Container Request to NM for AM container.
> But at INFO level from RM log not able find why RM is killing AM when job is not killed manually.
> One thing found common on failed AM logs is -:
> org.apache.hadoop.yarn.state.InvalidStateTransitonException
> With with different.
> For e.g. One log says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_UPDATE at ASSIGNED
> {code}
> Whereas other logs says -:
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: JOB_COUNTER_UPDATE at ERROR
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira