You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Tamas Domok (Jira)" <ji...@apache.org> on 2023/01/20 12:44:00 UTC
[jira] [Updated] (YARN-11422) fs2cs failes to generate valid capacity-scheduler.xml
[ https://issues.apache.org/jira/browse/YARN-11422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Domok updated YARN-11422:
-------------------------------
Description:
Attachments:
- [^fair-scheduler-issue.xml]
- [^yarn-site.xml]
Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml.
Run the following command:
{code}
rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp/output
{code}
The conversion fails with the following error:
{code}
2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
Suppressed: java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
... 2 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798)
... 5 more
Caused by: java.lang.ClassNotFoundException: fair
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049)
... 12 more
2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues
Verification failed: Failed to initialize queues
2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details!
{code}
With the following manual changes on the generated capacity-scheduler.xml:
- add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.users.ordering-policy=fair
RM can start up.
was:
Attachments:
- [^fair-scheduler-issue.xml]
- [^yarn-site.xml]
Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml.
Run the following command:
{code}
rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp/output
{code}
The conversion fails with the following error:
{code}
2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler
org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
Suppressed: java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
... 2 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798)
... 5 more
Caused by: java.lang.ClassNotFoundException: fair
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049)
... 12 more
2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues
Verification failed: Failed to initialize queues
2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details!
{code}
With the following manual changes on the generated capacity-scheduler.xml:
- add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true
- remove: yarn.scheduler.capacity.root.users.ordering-policy=fair
RM can start up.
> fs2cs failes to generate valid capacity-scheduler.xml
> -----------------------------------------------------
>
> Key: YARN-11422
> URL: https://issues.apache.org/jira/browse/YARN-11422
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Affects Versions: 3.4.0
> Reporter: Tamas Domok
> Assignee: Tamas Domok
> Priority: Major
> Attachments: fair-scheduler-issue.xml, yarn-site.xml
>
>
> Attachments:
> - [^fair-scheduler-issue.xml]
> - [^yarn-site.xml]
> Save them to /tmp/fair-scheduler.xml and /tmp/yarn-site.xml.
> Run the following command:
> {code}
> rm -rf /tmp/output/*; yarn fs2cs --cluster-resource memory-mb=36201037824,vcores=4914 --no-terminal-rule-check -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp/output
> {code}
> The conversion fails with the following error:
> {code}
> 2023-01-20 13:37:42,250 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state STOPPED
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
> at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
> 2023-01-20 13:37:42,251 ERROR converter.ConvertedConfigValidator: Could not start Capacity Scheduler
> org.apache.hadoop.yarn.exceptions.YarnException: Failed to initialize queues
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:807)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:318)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:412)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:70)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigArgumentHandler.parseAndConvert(FSConfigToCSConfigArgumentHandler.java:179)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverterMain.main(FSConfigToCSConfigConverterMain.java:47)
> Suppressed: java.lang.NullPointerException
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceStop(CapacityScheduler.java:428)
> at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at org.apache.hadoop.service.AbstractService.close(AbstractService.java:248)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.ConvertedConfigValidator.validateConvertedConfig(ConvertedConfigValidator.java:74)
> ... 2 more
> Caused by: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Unable to construct queue ordering policy=fair queue=root.default
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2053)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:172)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:140)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:115)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:276)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.parseQueue(CapacitySchedulerQueueManager.java:283)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.initializeQueues(CapacitySchedulerQueueManager.java:176)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:798)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException: fair
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getQueueOrderingPolicy(CapacitySchedulerConfiguration.java:2049)
> ... 12 more
> 2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigArgumentHandler: Verification failed: Failed to initialize queues
> Verification failed: Failed to initialize queues
> 2023-01-20 13:37:42,251 ERROR converter.FSConfigToCSConfigConverterMain: Error while starting FS configuration conversion, see previous error messages for details!
> {code}
> With the following manual changes on the generated capacity-scheduler.xml:
> - add: yarn.scheduler.capacity.root.users.auto-queue-creation-v2.enabled=true
> - remove: yarn.scheduler.capacity.root.default.auto-queue-creation-v2.enabled=true
> - remove: yarn.scheduler.capacity.root.users.ordering-policy=fair
> RM can start up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org