You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Tamas Domok (Jira)" <ji...@apache.org> on 2023/01/20 12:44:00 UTC
[jira] [Created] (YARN-11422) fs2cs failes to generate valid capacity-scheduler.xml
Tamas Domok created YARN-11422:
----------------------------------
Summary: fs2cs failes to generate valid capacity-scheduler.xml
Key: YARN-11422
URL: https://issues.apache.org/jira/browse/YARN-11422
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 3.4.0
Reporter: Tamas Domok
Assignee: Tamas Domok
Attachments: fair-scheduler-issue.xml, yarn-site.xml
Attachments:
- [^fair-scheduler-issue.xml]
- [^yarn-site.xml]
Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml.
Run the following command:
{code}
rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp/output
{code}
The conversion fails with the following error:
{code}
2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
Suppressed: java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
... 2 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798)
... 5 more
Caused by: java.lang.ClassNotFoundException: fair
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049)
... 12 more
2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues
Verification failed: Failed to initialize queues
2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details!
{code}
With the following manual changes on the generated capacity-scheduler.xml:
- add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.users.ordering-policy=fair
RM can start up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org