Capacity scheduler one queue per user

["Ratnasamy, Fritz" <fr...@chicagobooth.edu>]

Hello,

I am trying to administrate a Spark Hadoop CLuster with the open source
Hadoop distribution.
We would like to set up a Capacity Scheduler  with one queue per each user.
My yarn-site.xml has the following lines:

<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
   </property>


And This is my capacity-scheduler.xml


<!--  Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.  You
may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0  Unless required by
applicable law or agreed to in writing, software  distributed under
the License is distributed on an "AS IS" BASIS,  WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied.  See the License
for the specific language governing permissions and  limitations under
the License. See accompanying LICENSE file.--><configuration>
<property>    <name>yarn.scheduler.capacity.maximum-applications</name>
   <value>10000</value>    <description>      Maximum number of
applications that can be pending and running.    </description>
</property>  <property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.1</value>    <description>      Maximum percent of resources
in the cluster which can be used to run       application masters i.e.
controls number of concurrent running      applications.
</description>  </property>  <property>
<name>yarn.scheduler.capacity.root.default.capacity</name>
<value>60</value>  </property>   <property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
   <description>      The ResourceCalculator implementation to be used
to compare       Resources in the scheduler.      The default i.e.
DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.    </description>
 </property>  <property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>admin,users</value>    <description>      The queues at the
this level (root is the root queue).    </description>  </property>
<property>     <name>yarn.scheduler.capacity.root.users.auto-create-child-queue.enabled</name>
    <value>true</value>     <description>      Allow auto creation of
leaf queue.    </description>  </property>     <property>
<name>yarn.scheduler.capacity.root.admin.capacity</name>
<value>10</value>    <description>      The queues at the this level
is for admin (10% usage of the cluster).    </description>
</property>  <property>
<name>yarn.scheduler.capacity.root.users.capacity</name>
<value>90</value>    <description>      The queues at the this level
is for users (90% usage of the cluster)    </description>  </property>
   <property>    <name>yarn.scheduler.capacity.queue-mappings</name>
 <value>u:fratnasamy:admin</value>    <description>Specify user who
can submit to admin queue</description> </property>  <property>
<name>yarn.scheduler.capacity.queue-mappings</name>
<value>u:%user:users.%user</value>    <description>Maps users to
queues with the same name as user</description> </property>
<property>    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
   <value>false</value>    <description>      If a queue mapping is
present, will it override the value specified      by the user? This
can be used by administrators to place jobs in queues      that are
different than the one specified by the user.      The default is
false.    </description>  </property>  <property>
<name>yarn.scheduler.capacity.application.fail-fast</name>
<value>false</value>    <description>      Whether RM should fail
during recovery if previous applications'      queue is no longer
valid.    </description>  </property></configuration>

However, when i try to run the pi.py example (or any other examples), I run
into this error message : Exception in thread "main"
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1632944726077_0002 to YARN :
org.apache.hadoop.security.AccessControlException: Queue
root.users.fratnasamy already has 0 applications, Would anyone know how to
troubleshoot this issue?

*Fritz Ratnasamy*

Data Scientist

Information Technology

The University of Chicago

Booth School of Business

5807 S. Woodlawn

Chicago, Illinois 60637

Phone: +(1) 773-834-4556