You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Peter Bacsko (Jira)" <ji...@apache.org> on 2020/01/13 15:34:00 UTC

[jira] [Created] (YARN-10082) FS-CS converter: disable terminal placement rule checking

Peter Bacsko created YARN-10082:
-----------------------------------

             Summary: FS-CS converter: disable terminal placement rule checking
                 Key: YARN-10082
                 URL: https://issues.apache.org/jira/browse/YARN-10082
             Project: Hadoop YARN
          Issue Type: Sub-task
            Reporter: Peter Bacsko
            Assignee: Peter Bacsko


Before YARN-8967, {{QueuePlacementRule}} class had a method called {{isTerminal()}}. However, sometimes this method was hard-coded to return false, accepting such configurations as:

{noformat}
    <queuePlacementPolicy>
        <rule name="specified" create="true"/>
        <rule name="nestedUserQueue" create="true">
            <rule name="default" create="true" queue="users"/>
        </rule>
        <rule name="default"/>
    </queuePlacementPolicy>
{noformat}

It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.

This changed after YARN-8967, the behavior is different. Now, this configuration is not accepted because {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules differently:

https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183

In order to migrate existing configuration that were created before YARN-8967, we need a new switch (at least in migration mode) in FS to turn off this validation, otherwise the tool will not be able to migrate these configs and the following exception will be thrown:

{noformat}
~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion rules file is not defined, using default conversion config!
20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using explicitly defined fair-scheduler.xml
WARNING: This feature is experimental and not intended for production use!
20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/01/13 05:48:21 INFO security.YarnAuthorizationProvider: org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation = <memory:1024, vCores:1>
20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation = <memory:8192, vCores:2>
20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation file file:/tmp/fair-scheduler.xml
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule implementation: class org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
20/01/13 05:48:22 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler failed in state INITED
java.io.IOException: Failed to initialize FairScheduler
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
... 6 more
20/01/13 05:48:22 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion!
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Failed to initialize FairScheduler
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:116)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:44)
Caused by: java.io.IOException: Failed to initialize FairScheduler
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
... 4 more
Caused by: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationConfigurationException: Rules after rule 2 in queue placement policy can never be reached
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.updateRuleSet(QueuePlacementPolicy.java:110)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.fromXml(QueuePlacementPolicy.java:187)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.getQueuePlacementPolicy(AllocationFileLoaderService.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService.reloadAllocations(AllocationFileLoaderService.java:257)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1436)
... 6 more
{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org