[jira] [Commented] (YARN-3843) Fair Scheduler should not accept apps with space keys as queue name

["Dongwook Kwon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14596842#comment-14596842 ] 

Dongwook Kwon commented on YARN-3843:
-------------------------------------

>From my investigation, QueueMetrics doesn't allow space key string as start or end of names, it just trims empty strings.

static final Splitter Q_SPLITTER = Splitter.on('.').omitEmptyStrings().trimResults();

https://github.com/apache/hadoop/blob/branch-2.5.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java#L112
https://github.com/apache/hadoop/blob/branch-2.5.2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java#L85

So, from FairScheduler, "root.adhoc.birvine ", this queue name with the space at the end of name, it is treated as different from "root.adhoc.birvine" because it has one more character, and from QueueMetrics, because names are trimmed, all of sudden, 2 different queue names become the same that causes the error as "Metrics source QueueMetrics,q0=root,q1=adhoc,q2=birvine already exists!"


> Fair Scheduler should not accept apps with space keys as queue name
> -------------------------------------------------------------------
>
>                 Key: YARN-3843
>                 URL: https://issues.apache.org/jira/browse/YARN-3843
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.4.0, 2.5.0
>            Reporter: Dongwook Kwon
>            Priority: Minor
>
> As YARN-461, since empty string queue name is not valid, queue name with space keys such as " " ,"   " should not be accepted either, also not as prefix nor postfix. 
> e.g) "root.test.queuename  ", or "root.test. queuename"
> I have 2 specific cases kill RM with these space keys as part of queue name.
> 1) Without placement policy (hadoop 2.4.0 and above), 
> When a job is submitted with " "(space key) as queue name
> e.g) mapreduce.job.queuename=" "
> 2) With placement policy (hadoop 2.5.0 and above)
>  Once a job is submitted without space key as queue name, and submit another job with space key.
> e.g) 1st time: mapreduce.job.queuename="root.test.user1" 
> 2nd time: mapreduce.job.queuename="root.test.user1 "
> {code}
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.974 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
> testQueueNameWithSpace(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)  Time elapsed: 0.724 sec  <<< ERROR!
> org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=adhoc,q2=birvine already exists!
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:135)
> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:112)
> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:218)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueueMetrics.forQueue(FSQueueMetrics.java:96)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.<init>(FSQueue.java:56)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.<init>(FSLeafQueue.java:66)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.createQueue(QueueManager.java:169)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getQueue(QueueManager.java:120)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.getLeafQueue(QueueManager.java:88)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:660)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:569)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1127)
> 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testQueueNameWithSpace(TestFairScheduler.java:627)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)