You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by #TANG SHANJIANG# <ST...@e.ntu.edu.sg> on 2013/12/06 04:05:46 UTC

MapReduce Job running Problems with queue designation in fair scheduler for Yarn-2.2.0.

Hi,
   I encounter a problem with Yarn's fair scheduler. The thing is that,  I first set a queue by configuring fair-scheduler.xml below. Next I try to submit a job to that queue by designating queue name via "mapreduce.job.queuename= amelie".
fair-scheduler.xml:
<allocations>
  <queue name="amelie">
    <minResources>10000 mb,1vcores</minResources>
    <maxResources>90000 mb,1vcores</maxResources>
    <maxRunningApps>50</maxRunningApps>
    <weight>2.0</weight>
    <schedulingPolicy>fair</schedulingPolicy>
  </queue>
</allocations>
However, the job is stuck in that queue without running, and the yarn-amelie-resourcemanager-pdcbl3-log file shows  errors that,
2013-12-06 10:16:20,551 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:amelie (auth:SIMPLE) cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1386295642784_0001' doesn't exist in RM.
2013-12-06 10:16:20,552 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 8032, call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport from 155.69.148.21:60571 Call#1416 Retry#0: error: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1386295642784_0001' doesn't exist in RM.
org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1386295642784_0001' doesn't exist in RM.
        at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:247)
        at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:120)
        at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:241)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)

Moreover, the strange point is that, if I don't set the queue 'amelie' in fair-scheduler.xml and run my job directly. The problem can run successfully by creating a queue called 'amelie' automatically. It seems that Yarn's fair scheduler do not support the job running on user's queue configuration in fair-scheduler.xml

Re: MapReduce Job running Problems with queue designation in fair scheduler for Yarn-2.2.0.

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi Tang,

One concerning thing is that the vcores for amelie's maxResources is set to
1.  Not sure how that would lead to the error you're seeing, but it would
prevent applications from running in that queue.  Is your job able to run
if you set this to someting higher?

-Sandy


On Thu, Dec 5, 2013 at 7:05 PM, #TANG SHANJIANG# <ST...@e.ntu.edu.sg>wrote:

>  Hi,
>
>    I encounter a problem with Yarn’s fair scheduler. The thing is that,  I
> first set a queue by configuring fair-scheduler.xml below. Next I try to
> submit a job to that queue by designating queue name via
> “mapreduce.job.queuename=* amelie*”.
>
> *fair-scheduler.xml:*
>
> <allocations>
>
>   <queue name="amelie">
>
>     <minResources>10000 mb,1vcores</minResources>
>
>     <maxResources>90000 mb,1vcores</maxResources>
>
>     <maxRunningApps>50</maxRunningApps>
>
>     <weight>2.0</weight>
>
>     <schedulingPolicy>fair</schedulingPolicy>
>
>   </queue>
>
> </allocations>
>
> *However, the job is stuck in that queue without running, and the
> yarn-amelie-resourcemanager-pdcbl3-log file shows  errors that, *
>
> 2013-12-06 10:16:20,551 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:amelie (auth:SIMPLE)
> cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
> 2013-12-06 10:16:20,552 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 8032, call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
> from 155.69.148.21:60571 Call#1416 Retry#0: error:
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1386295642784_0001' doesn't exist in RM.
>
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
>         at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:247)
>
>         at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:120)
>
>         at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:241)
>
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> *Moreover, the strange point is that, if I don’t set the queue ‘amelie’ in
> fair-scheduler.xml and run my job directly. The problem can run
> successfully by creating a queue called ‘amelie’ automatically. It seems
> that Yarn’s fair scheduler do not support the job running on user’s queue
> configuration in fair-scheduler.xml*
>
>
>

Re: MapReduce Job running Problems with queue designation in fair scheduler for Yarn-2.2.0.

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi Tang,

One concerning thing is that the vcores for amelie's maxResources is set to
1.  Not sure how that would lead to the error you're seeing, but it would
prevent applications from running in that queue.  Is your job able to run
if you set this to someting higher?

-Sandy


On Thu, Dec 5, 2013 at 7:05 PM, #TANG SHANJIANG# <ST...@e.ntu.edu.sg>wrote:

>  Hi,
>
>    I encounter a problem with Yarn’s fair scheduler. The thing is that,  I
> first set a queue by configuring fair-scheduler.xml below. Next I try to
> submit a job to that queue by designating queue name via
> “mapreduce.job.queuename=* amelie*”.
>
> *fair-scheduler.xml:*
>
> <allocations>
>
>   <queue name="amelie">
>
>     <minResources>10000 mb,1vcores</minResources>
>
>     <maxResources>90000 mb,1vcores</maxResources>
>
>     <maxRunningApps>50</maxRunningApps>
>
>     <weight>2.0</weight>
>
>     <schedulingPolicy>fair</schedulingPolicy>
>
>   </queue>
>
> </allocations>
>
> *However, the job is stuck in that queue without running, and the
> yarn-amelie-resourcemanager-pdcbl3-log file shows  errors that, *
>
> 2013-12-06 10:16:20,551 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:amelie (auth:SIMPLE)
> cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
> 2013-12-06 10:16:20,552 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 8032, call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
> from 155.69.148.21:60571 Call#1416 Retry#0: error:
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1386295642784_0001' doesn't exist in RM.
>
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
>         at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:247)
>
>         at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:120)
>
>         at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:241)
>
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> *Moreover, the strange point is that, if I don’t set the queue ‘amelie’ in
> fair-scheduler.xml and run my job directly. The problem can run
> successfully by creating a queue called ‘amelie’ automatically. It seems
> that Yarn’s fair scheduler do not support the job running on user’s queue
> configuration in fair-scheduler.xml*
>
>
>

Re: MapReduce Job running Problems with queue designation in fair scheduler for Yarn-2.2.0.

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi Tang,

One concerning thing is that the vcores for amelie's maxResources is set to
1.  Not sure how that would lead to the error you're seeing, but it would
prevent applications from running in that queue.  Is your job able to run
if you set this to someting higher?

-Sandy


On Thu, Dec 5, 2013 at 7:05 PM, #TANG SHANJIANG# <ST...@e.ntu.edu.sg>wrote:

>  Hi,
>
>    I encounter a problem with Yarn’s fair scheduler. The thing is that,  I
> first set a queue by configuring fair-scheduler.xml below. Next I try to
> submit a job to that queue by designating queue name via
> “mapreduce.job.queuename=* amelie*”.
>
> *fair-scheduler.xml:*
>
> <allocations>
>
>   <queue name="amelie">
>
>     <minResources>10000 mb,1vcores</minResources>
>
>     <maxResources>90000 mb,1vcores</maxResources>
>
>     <maxRunningApps>50</maxRunningApps>
>
>     <weight>2.0</weight>
>
>     <schedulingPolicy>fair</schedulingPolicy>
>
>   </queue>
>
> </allocations>
>
> *However, the job is stuck in that queue without running, and the
> yarn-amelie-resourcemanager-pdcbl3-log file shows  errors that, *
>
> 2013-12-06 10:16:20,551 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:amelie (auth:SIMPLE)
> cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
> 2013-12-06 10:16:20,552 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 8032, call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
> from 155.69.148.21:60571 Call#1416 Retry#0: error:
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1386295642784_0001' doesn't exist in RM.
>
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
>         at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:247)
>
>         at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:120)
>
>         at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:241)
>
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> *Moreover, the strange point is that, if I don’t set the queue ‘amelie’ in
> fair-scheduler.xml and run my job directly. The problem can run
> successfully by creating a queue called ‘amelie’ automatically. It seems
> that Yarn’s fair scheduler do not support the job running on user’s queue
> configuration in fair-scheduler.xml*
>
>
>

Re: MapReduce Job running Problems with queue designation in fair scheduler for Yarn-2.2.0.

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi Tang,

One concerning thing is that the vcores for amelie's maxResources is set to
1.  Not sure how that would lead to the error you're seeing, but it would
prevent applications from running in that queue.  Is your job able to run
if you set this to someting higher?

-Sandy


On Thu, Dec 5, 2013 at 7:05 PM, #TANG SHANJIANG# <ST...@e.ntu.edu.sg>wrote:

>  Hi,
>
>    I encounter a problem with Yarn’s fair scheduler. The thing is that,  I
> first set a queue by configuring fair-scheduler.xml below. Next I try to
> submit a job to that queue by designating queue name via
> “mapreduce.job.queuename=* amelie*”.
>
> *fair-scheduler.xml:*
>
> <allocations>
>
>   <queue name="amelie">
>
>     <minResources>10000 mb,1vcores</minResources>
>
>     <maxResources>90000 mb,1vcores</maxResources>
>
>     <maxRunningApps>50</maxRunningApps>
>
>     <weight>2.0</weight>
>
>     <schedulingPolicy>fair</schedulingPolicy>
>
>   </queue>
>
> </allocations>
>
> *However, the job is stuck in that queue without running, and the
> yarn-amelie-resourcemanager-pdcbl3-log file shows  errors that, *
>
> 2013-12-06 10:16:20,551 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:amelie (auth:SIMPLE)
> cause:org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
> 2013-12-06 10:16:20,552 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 4 on 8032, call
> org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport
> from 155.69.148.21:60571 Call#1416 Retry#0: error:
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application
> with id 'application_1386295642784_0001' doesn't exist in RM.
>
> org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException:
> Application with id 'application_1386295642784_0001' doesn't exist in RM.
>
>         at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:247)
>
>         at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:120)
>
>         at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:241)
>
>         at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
>
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)
>
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)
>
>
>
> *Moreover, the strange point is that, if I don’t set the queue ‘amelie’ in
> fair-scheduler.xml and run my job directly. The problem can run
> successfully by creating a queue called ‘amelie’ automatically. It seems
> that Yarn’s fair scheduler do not support the job running on user’s queue
> configuration in fair-scheduler.xml*
>
>
>