You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jay Vyas <ja...@gmail.com> on 2014/04/24 22:23:23 UTC
Yarn hangs @Scheduled
Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've
restarted my nodemanager a few times , but no luck.
What are the possible reasons that YARN job submission hangs ? I know one
is resource availability, but this is a fresh cluster on a VM with only one
job, one NM, and one RM.
14/04/24 16:20:32 INFO ipc.Server: Auth successful for
yarn@IDH1.LOCAL(auth:SIMPLE)
14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager: Authorization
successful for yarn@IDH1.LOCAL (auth:KERBEROS) for protocol=interface
org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new
applicationId: 4
14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with id
4 submitted by user yarn
14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn
IP=192.168.122.100 OPERATION=Submit Application Request
TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id
application_1398370674313_0004
14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
State change from NEW to NEW_SAVING
14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app:
application_1398370674313_0004
14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
State change from NEW_SAVING to SUBMITTED
14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application
application_1398370674313_0004 from user: yarn, in queue: default,
currently num of applications: 4
14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
State change from SUBMITTED to ACCEPTED
14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService:
Registering app attempt : appattempt_1398370674313_0004_000001
14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt
appattempt_1398370674313_0004_000001 to scheduler from user: yarn
14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
appattempt_1398370674313_0004_000001 State change from SUBMITTED to
SCHEDULED
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Yarn hangs @Scheduled
Posted by Jay Vyas <ja...@gmail.com>.
I fixed the issue by setting
yarn.scheduler.minimum-allocation-mb=1024
I'm thinking this happens a lot in VMs where you run w low memory.
If memory too low, I think other failures will occur at runtime when you start daemons or tasks...If too high, then the tasks will hang...
> On Apr 24, 2014, at 5:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> How much memory do you see as available on the RM web page? And what are the memory requirements for this app? And this is a MR job?
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>> On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've restarted my nodemanager a few times , but no luck.
>>
>> What are the possible reasons that YARN job submission hangs ? I know one is resource availability, but this is a fresh cluster on a VM with only one job, one NM, and one RM.
>>
>> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL (auth:SIMPLE)
>> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager: Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
>> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new applicationId: 4
>> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with id 4 submitted by user yarn
>> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn IP=192.168.122.100 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW to NEW_SAVING
>> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app: application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW_SAVING to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application application_1398370674313_0004 from user: yarn, in queue: default, currently num of applications: 4
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from SUBMITTED to ACCEPTED
>> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1398370674313_0004_000001
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt appattempt_1398370674313_0004_000001 to scheduler from user: yarn
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from SUBMITTED to SCHEDULED
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Jay Vyas <ja...@gmail.com>.
I fixed the issue by setting
yarn.scheduler.minimum-allocation-mb=1024
I'm thinking this happens a lot in VMs where you run w low memory.
If memory too low, I think other failures will occur at runtime when you start daemons or tasks...If too high, then the tasks will hang...
> On Apr 24, 2014, at 5:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> How much memory do you see as available on the RM web page? And what are the memory requirements for this app? And this is a MR job?
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>> On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've restarted my nodemanager a few times , but no luck.
>>
>> What are the possible reasons that YARN job submission hangs ? I know one is resource availability, but this is a fresh cluster on a VM with only one job, one NM, and one RM.
>>
>> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL (auth:SIMPLE)
>> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager: Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
>> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new applicationId: 4
>> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with id 4 submitted by user yarn
>> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn IP=192.168.122.100 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW to NEW_SAVING
>> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app: application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW_SAVING to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application application_1398370674313_0004 from user: yarn, in queue: default, currently num of applications: 4
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from SUBMITTED to ACCEPTED
>> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1398370674313_0004_000001
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt appattempt_1398370674313_0004_000001 to scheduler from user: yarn
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from SUBMITTED to SCHEDULED
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Jay Vyas <ja...@gmail.com>.
I fixed the issue by setting
yarn.scheduler.minimum-allocation-mb=1024
I'm thinking this happens a lot in VMs where you run w low memory.
If memory too low, I think other failures will occur at runtime when you start daemons or tasks...If too high, then the tasks will hang...
> On Apr 24, 2014, at 5:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> How much memory do you see as available on the RM web page? And what are the memory requirements for this app? And this is a MR job?
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>> On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've restarted my nodemanager a few times , but no luck.
>>
>> What are the possible reasons that YARN job submission hangs ? I know one is resource availability, but this is a fresh cluster on a VM with only one job, one NM, and one RM.
>>
>> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL (auth:SIMPLE)
>> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager: Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
>> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new applicationId: 4
>> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with id 4 submitted by user yarn
>> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn IP=192.168.122.100 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW to NEW_SAVING
>> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app: application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW_SAVING to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application application_1398370674313_0004 from user: yarn, in queue: default, currently num of applications: 4
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from SUBMITTED to ACCEPTED
>> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1398370674313_0004_000001
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt appattempt_1398370674313_0004_000001 to scheduler from user: yarn
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from SUBMITTED to SCHEDULED
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Jay Vyas <ja...@gmail.com>.
I fixed the issue by setting
yarn.scheduler.minimum-allocation-mb=1024
I'm thinking this happens a lot in VMs where you run w low memory.
If memory too low, I think other failures will occur at runtime when you start daemons or tasks...If too high, then the tasks will hang...
> On Apr 24, 2014, at 5:25 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
> How much memory do you see as available on the RM web page? And what are the memory requirements for this app? And this is a MR job?
>
> +Vinod
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>> On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've restarted my nodemanager a few times , but no luck.
>>
>> What are the possible reasons that YARN job submission hangs ? I know one is resource availability, but this is a fresh cluster on a VM with only one job, one NM, and one RM.
>>
>> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL (auth:SIMPLE)
>> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager: Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
>> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new applicationId: 4
>> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with id 4 submitted by user yarn
>> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn IP=192.168.122.100 OPERATION=Submit Application Request TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW to NEW_SAVING
>> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app: application_1398370674313_0004
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from NEW_SAVING to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application application_1398370674313_0004 from user: yarn, in queue: default, currently num of applications: 4
>> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004 State change from SUBMITTED to ACCEPTED
>> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1398370674313_0004_000001
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
>> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt appattempt_1398370674313_0004_000001 to scheduler from user: yarn
>> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl: appattempt_1398370674313_0004_000001 State change from SUBMITTED to SCHEDULED
>>
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
How much memory do you see as available on the RM web page? And what are
the memory requirements for this app? And this is a MR job?
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've
> restarted my nodemanager a few times , but no luck.
>
> What are the possible reasons that YARN job submission hangs ? I know one
> is resource availability, but this is a fresh cluster on a VM with only one
> job, one NM, and one RM.
>
> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL(auth:SIMPLE)
> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager:
> Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for
> protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new
> applicationId: 4
> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with
> id 4 submitted by user yarn
> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn
> IP=192.168.122.100 OPERATION=Submit Application Request
> TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW to NEW_SAVING
> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app:
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW_SAVING to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application
> application_1398370674313_0004 from user: yarn, in queue: default,
> currently num of applications: 4
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from SUBMITTED to ACCEPTED
> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService:
> Registering app attempt : appattempt_1398370674313_0004_000001
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt
> appattempt_1398370674313_0004_000001 to scheduler from user: yarn
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from SUBMITTED to
> SCHEDULED
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
How much memory do you see as available on the RM web page? And what are
the memory requirements for this app? And this is a MR job?
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've
> restarted my nodemanager a few times , but no luck.
>
> What are the possible reasons that YARN job submission hangs ? I know one
> is resource availability, but this is a fresh cluster on a VM with only one
> job, one NM, and one RM.
>
> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL(auth:SIMPLE)
> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager:
> Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for
> protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new
> applicationId: 4
> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with
> id 4 submitted by user yarn
> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn
> IP=192.168.122.100 OPERATION=Submit Application Request
> TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW to NEW_SAVING
> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app:
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW_SAVING to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application
> application_1398370674313_0004 from user: yarn, in queue: default,
> currently num of applications: 4
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from SUBMITTED to ACCEPTED
> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService:
> Registering app attempt : appattempt_1398370674313_0004_000001
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt
> appattempt_1398370674313_0004_000001 to scheduler from user: yarn
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from SUBMITTED to
> SCHEDULED
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
How much memory do you see as available on the RM web page? And what are
the memory requirements for this app? And this is a MR job?
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've
> restarted my nodemanager a few times , but no luck.
>
> What are the possible reasons that YARN job submission hangs ? I know one
> is resource availability, but this is a fresh cluster on a VM with only one
> job, one NM, and one RM.
>
> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL(auth:SIMPLE)
> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager:
> Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for
> protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new
> applicationId: 4
> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with
> id 4 submitted by user yarn
> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn
> IP=192.168.122.100 OPERATION=Submit Application Request
> TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW to NEW_SAVING
> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app:
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW_SAVING to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application
> application_1398370674313_0004 from user: yarn, in queue: default,
> currently num of applications: 4
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from SUBMITTED to ACCEPTED
> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService:
> Registering app attempt : appattempt_1398370674313_0004_000001
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt
> appattempt_1398370674313_0004_000001 to scheduler from user: yarn
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from SUBMITTED to
> SCHEDULED
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Yarn hangs @Scheduled
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
How much memory do you see as available on the RM web page? And what are
the memory requirements for this app? And this is a MR job?
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Thu, Apr 24, 2014 at 1:23 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi folks : My yarn jobs seem to be hanging in the "SHEDULED" state. I've
> restarted my nodemanager a few times , but no luck.
>
> What are the possible reasons that YARN job submission hangs ? I know one
> is resource availability, but this is a fresh cluster on a VM with only one
> job, one NM, and one RM.
>
> 14/04/24 16:20:32 INFO ipc.Server: Auth successful for yarn@IDH1.LOCAL(auth:SIMPLE)
> 14/04/24 16:20:32 INFO authorize.ServiceAuthorizationManager:
> Authorization successful for yarn@IDH1.LOCAL (auth:KERBEROS) for
> protocol=interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB
> 14/04/24 16:20:32 INFO resourcemanager.ClientRMService: Allocated new
> applicationId: 4
> 14/04/24 16:20:33 INFO resourcemanager.ClientRMService: Application with
> id 4 submitted by user yarn
> 14/04/24 16:20:33 INFO resourcemanager.RMAuditLogger: USER=yarn
> IP=192.168.122.100 OPERATION=Submit Application Request
> TARGET=ClientRMService RESULT=SUCCESS APPID=application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: Storing application with id
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW to NEW_SAVING
> 14/04/24 16:20:33 INFO recovery.RMStateStore: Storing info for app:
> application_1398370674313_0004
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from NEW_SAVING to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Accepted application
> application_1398370674313_0004 from user: yarn, in queue: default,
> currently num of applications: 4
> 14/04/24 16:20:33 INFO rmapp.RMAppImpl: application_1398370674313_0004
> State change from SUBMITTED to ACCEPTED
> 14/04/24 16:20:33 INFO resourcemanager.ApplicationMasterService:
> Registering app attempt : appattempt_1398370674313_0004_000001
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from NEW to SUBMITTED
> 14/04/24 16:20:33 INFO fair.FairScheduler: Added Application Attempt
> appattempt_1398370674313_0004_000001 to scheduler from user: yarn
> 14/04/24 16:20:33 INFO attempt.RMAppAttemptImpl:
> appattempt_1398370674313_0004_000001 State change from SUBMITTED to
> SCHEDULED
>
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.