You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Peter Bacsko (Jira)" <ji...@apache.org> on 2020/01/13 15:34:00 UTC
[jira] [Created] (YARN-10082) FS-CS converter: disable terminal
placement rule checking
Peter Bacsko created YARN-10082:
-----------------------------------
Summary: FS-CS converter: disable terminal placement rule checking
Key: YARN-10082
URL: https://issues.apache.org/jira/browse/YARN-10082
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Peter Bacsko
Assignee: Peter Bacsko
Before YARN-8967, {{QueuePlacementRule}} class had a method called {{isTerminal()}}. However, sometimes this method was hard-coded to return false, accepting such configurations as:
{noformat}
<queuePlacementPolicy>
<rule name="specified" create="true"/>
<rule name="nestedUserQueue" create="true">
<rule name="default" create="true" queue="users"/>
</rule>
<rule name="default"/>
</queuePlacementPolicy>
{noformat}
It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
This changed after YARN-8967, the behavior is different. Now, this configuration is not accepted because {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules differently:
https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
In order to migrate existing configuration that were created before YARN-8967, we need a new switch (at least in migration mode) in FS to turn off this validation, otherwise the tool will not be able to migrate these configs and the following exception will be thrown:
{noformat}
~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion rules file is not defined, using default conversion config!
20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using explicitly defined fair-scheduler.xml
WARNING: This feature is experimental and not intended for production use!
20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/01/13 05:48:21 INFO security.YarnAuthorizationProvider: org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation = <memory:1024, vCores:1>
20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation = <memory:8192, vCores:2>
20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation file file:/tmp/fair-scheduler.xml
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
20/01/13 05:48:22 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED
java.io.IOException: Failed to initialize FairScheduler
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
... 6 more
20/01/13 05:48:22 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion!
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Failed to initialize FairScheduler
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
Caused by: java.io.IOException: Failed to initialize FairScheduler
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
... 4 more
Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
... 6 more
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org