You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "shameersss1 (via GitHub)" <gi...@apache.org> on 2024/10/31 12:02:26 UTC
[PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
shameersss1 opened a new pull request, #7138:
URL: https://github.com/apache/hadoop/pull/7138
### Description of PR
Enable asynchronous scheduling by default for capacity scheduler
### How was this patch tested?
### For code changes:
- [x ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7327: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 merged PR #7138:
URL: https://github.com/apache/hadoop/pull/7138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "zeekling (via GitHub)" <gi...@apache.org>.
zeekling commented on code in PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#discussion_r1846910583
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java:
##########
@@ -927,7 +927,11 @@ public void testReleaseOutdatedReservedContainer() throws Exception {
* First proposal should be accepted, second proposal should be rejected
* because it try to release an outdated reserved container
*/
- MockRM rm1 = new MockRM();
+ // disable async-scheduling for simulating complex scene
+ Configuration disableAsyncConf = new Configuration(conf);
+ disableAsyncConf.setBoolean(
+ CapacitySchedulerConfiguration.SCHEDULE_ASYNCHRONOUSLY_ENABLE, false);
Review Comment:
Why disable async scheduler in UT? And enable by default? I think we'd better enable async schedule by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7327: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "brumi1024 (via GitHub)" <gi...@apache.org>.
brumi1024 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2551713490
Thanks @shameersss1 for the change, @TaoYang526 @zeekling @slfan1989 for the reviews, merged to trunk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2526655420
@shameersss1 Thank you very much for submitting the code! However, I am not very familiar with the Capacity Scheduler. To ensure the quality of the code, I will ask a member who is more familiar with the relevant code to help with the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7327: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2531512162
@brumi1024 - Thanks for looking into this.
> what is the reason behind changing the default of this setting?
1. The current default scheduling mechanism is synchronous (node-heart driven) which is not efficient when there are large number of containers to be allocated.
2. It also has additional issues like scheduling won't happen if there is node-heartbeat loss due to network issue .
3. @wangdatan did an amazing job of making the async scheudling production ready : Refer https://issues.apache.org/jira/browse/YARN-7327?focusedCommentId=16205259&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16205259 for benchmark details.
4. The above benchmark shows async scheudling throughput is better than sync scheduling
And hence the proposal here is to change the default scheduling stratergy for capacity scheduler from synchronous to asynchronous. Already companies like Alibaba cloud use this in their production https://www.alibabacloud.com/help/en/emr/emr-on-ecs/user-guide/yarn-schedulers
@brumi1024 - Do you think is there any blocker/issue in enabling it by default ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2454520349
@slfan1989 @aajisaka @TaoYang526 Could you please review the changes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7327: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "zeekling (via GitHub)" <gi...@apache.org>.
zeekling commented on code in PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#discussion_r1880390597
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java:
##########
@@ -927,7 +927,11 @@ public void testReleaseOutdatedReservedContainer() throws Exception {
* First proposal should be accepted, second proposal should be rejected
* because it try to release an outdated reserved container
*/
- MockRM rm1 = new MockRM();
+ // disable async-scheduling for simulating complex scene
+ Configuration disableAsyncConf = new Configuration(conf);
+ disableAsyncConf.setBoolean(
+ CapacitySchedulerConfiguration.SCHEDULE_ASYNCHRONOUSLY_ENABLE, false);
Review Comment:
ok,get it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2476155352
@slfan1989 - Gentle reminder for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7327: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2551591139
@shameersss1 Thank you for your contribution. From my perspective, +1. A big thanks to @brumi1024 for helping review the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2525648133
@Hexiaoqiao could you please review the changes ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on code in PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#discussion_r1848586005
##########
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAsyncScheduling.java:
##########
@@ -927,7 +927,11 @@ public void testReleaseOutdatedReservedContainer() throws Exception {
* First proposal should be accepted, second proposal should be rejected
* because it try to release an outdated reserved container
*/
- MockRM rm1 = new MockRM();
+ // disable async-scheduling for simulating complex scene
+ Configuration disableAsyncConf = new Configuration(conf);
+ disableAsyncConf.setBoolean(
+ CapacitySchedulerConfiguration.SCHEDULE_ASYNCHRONOUSLY_ENABLE, false);
Review Comment:
The following are the reasons why async scheduling is disabled in UT :
1. unlike async scheduling, sync scheduling gives more flexibility while writing UTs to mimic complex scenarios since the UT writer will have full control over when to schedule, where as with async scheduling this cannot be done.
2. There were 200+ failing when async scheduling was enabled (due to scheduling assertion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2526656811
@brumi1024 Could you please help review this PR? Thank you very much!
cc: @TaoYang526 @Hexiaoqiao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7327: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2550920952
@brumi1024 - Please let me know if there is any more concerns. Now that the dependent YARN issue is also merged, Can we take it forward ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "shameersss1 (via GitHub)" <gi...@apache.org>.
shameersss1 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2449686472
Submitted a first cut PR to trigger unit tests for Yarn capacity scheduler with async scheduling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2450476778
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 40s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 34m 11s | | trunk passed |
| +1 :green_heart: | compile | 8m 11s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 7m 23s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +0 :ok: | spotbugs | 0m 36s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 38m 49s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 7m 20s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 7m 20s | | the patch passed |
| +1 :green_heart: | compile | 7m 18s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 7m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 48s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 39m 36s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 159m 1s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 30s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not generate ASF License warnings. |
| | | 344m 36s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulingWithAllocationRequestId |
| | hadoop.yarn.server.resourcemanager.TestRMRestart |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSchedulingRequestUpdate |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocationAsync |
| | hadoop.yarn.server.resourcemanager.TestResourceManager |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption |
| | hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCSAllocateCustomResource |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivities |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerOvercommit |
| | hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService |
| | hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
| | hadoop.yarn.server.resourcemanager.TestRM |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimitsByPartition |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerApps |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodes |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWeightMode |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions |
| | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerMultiNodes |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerMultiNodesWithPreemption |
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesSchedulerActivitiesWithMultiNodesEnabled |
| | hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7138 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint |
| uname | Linux 054a961d9966 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b593898e81e40ab3767794305b998dcfb3f2ffe4 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/1/testReport/ |
| Max. process+thread count | 1393 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2453521451
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 7m 16s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 5 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 53s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 43s | | trunk passed |
| +1 :green_heart: | compile | 3m 43s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 3m 24s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 56s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +0 :ok: | spotbugs | 0m 24s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 19m 55s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 3m 20s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 3m 20s | | the patch passed |
| +1 :green_heart: | compile | 3m 21s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 3m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +0 :ok: | spotbugs | 0m 23s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 19m 51s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 90m 22s | [/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt) | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 22s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not generate ASF License warnings. |
| | | 200m 8s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7138 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux f10a7f02a241 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8f477e15f43319e3e240e4d6f2f69e251c0a4b0e |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/2/testReport/ |
| Max. process+thread count | 951 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "hadoop-yetus (via GitHub)" <gi...@apache.org>.
hadoop-yetus commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2454331301
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 6 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 56s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 37s | | trunk passed |
| +1 :green_heart: | compile | 8m 11s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 7m 16s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +0 :ok: | spotbugs | 0m 33s | | branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 39m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 7s | | the patch passed |
| +1 :green_heart: | compile | 7m 58s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 7m 58s | | the patch passed |
| +1 :green_heart: | compile | 7m 55s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 7m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 1m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +0 :ok: | spotbugs | 0m 28s | | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 39m 30s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 106m 54s | | hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m 30s | | hadoop-yarn-site in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not generate ASF License warnings. |
| | | 295m 6s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7138 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint markdownlint |
| uname | Linux 5087bdb3b28c 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0bb32d067009bdf93d651fb0b955258bc3fad646 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/3/testReport/ |
| Max. process+thread count | 950 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7138/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
Re: [PR] YARN-7237: Enable asynchronous scheduling by default for capacity scheduler [hadoop]
Posted by "TaoYang526 (via GitHub)" <gi...@apache.org>.
TaoYang526 commented on PR #7138:
URL: https://github.com/apache/hadoop/pull/7138#issuecomment-2455980699
@shameersss1 Thanks for your contribution.
The PR looks good to me. The improvement in document will be helpful for users, good job!
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org