You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Szilard Nemeth (JIRA)" <ji...@apache.org> on 2018/10/29 12:42:04 UTC
[jira] [Comment Edited] (YARN-5106) Provide a builder interface for
FairScheduler allocations for use in tests
[ https://issues.apache.org/jira/browse/YARN-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667138#comment-16667138 ]
Szilard Nemeth edited comment on YARN-5106 at 10/29/18 12:41 PM:
-----------------------------------------------------------------
About the test failures:
1. TestAMRMClient.testAMRMClientWithContainerResourceChange: seems to be unrelated (or flaky?)
2. TestContainerAllocation: The same set of testcases are failed on trunk and with my patch (local execution), so this is also unrelated.
3. TestAllocationFileLoaderService.testQueueNameContainingNBWhitespace: Passes for me after 15 runs of the whole class.
Added a @Before method that deletes the allocation file, which is a good practice on its own, regardless of the failure we experienced with the Jenkins results.
4. TestFairScheduler.testHierarchicalQueueAllocationFileParsing: Fixed the test, it was a trivial one as my patch did not add queueC and queueD as a children of queueA, so it was due to a difference in the structure of the allocation file.
5. TestFairScheduler.testFairShareAndWeightsInNestedUserQueueRule: This also generated changes in the allocation.xml file compared to the original code, so I fixed it.
However, the config my patch was generated with the testcases' code was invalid in a way that it procuded an NPE.
I created YARN-8951 for this to investigate further.
Notes for patch001:
I just did a simple find for all tests that generates allocation files (i.e. searched for "<allocations>") and replaced all of queue file creations with {{PrintWriter}}s with the queue config file builder introduced in YARN-7841.
There's one exception, though: TestFairSchedulerPreemption#writeAllocFile, this is because work is in progress on YARN-8059, so I wanted to avoid huge merge conflicts.
If YARN-8059 goes in first, I will rebase the patch and apply the changes to preemption test, otherwise I will create a separate jira to apply the changes to {{TestFairSchedulerPreemption}} as well.
was (Author: snemeth):
About the test failures:
1. TestAMRMClient.testAMRMClientWithContainerResourceChange: seems to be unrelated (or flaky?)
2. TestContainerAllocation: The same set of testcases are failed on trunk and with my patch (local execution)
3. TestAllocationFileLoaderService.testQueueNameContainingNBWhitespace: Passes for me after 15 runs of the whole class.
Added a @Before method that deletes the allocation file, which is a good practice on its own, regardless of the failure we experienced from the Jenkins run.
4. TestFairScheduler.testHierarchicalQueueAllocationFileParsing: Fixed the test, it was a trivial one as my patch did not add queueC and queueD as a children of queueA, so it was due to a difference in the structure of the allocation file.
5. TestFairScheduler.testFairShareAndWeightsInNestedUserQueueRule: This also generated changes in the allocation.xml file compared to the original code, so I fixed it.
However, the config my patch was generated with the testcases' code was invalid in a way that it procuded an NPE.
I created YARN-8951 for this to investigate further.
Notes for patch001:
I just did a simple find for all tests that generates allocation files (i.e. searched for "<allocations>") and replaced all of queue file creations with printwriters with the queue config file builder introduced in YARN-7841.
There's one exception, though: TestFairSchedulerPreemption#writeAllocFile, this is because work is in progress on YARN-8059, so I wanted to avoid huge merge conflicts.
If YARN-8059 goes in first, I will rebase the patch and apply the changes to preemption test, otherwise I will create a separate jira to apply the changes to \{{TestFairSchedulerPreemption}} as well.
> Provide a builder interface for FairScheduler allocations for use in tests
> --------------------------------------------------------------------------
>
> Key: YARN-5106
> URL: https://issues.apache.org/jira/browse/YARN-5106
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
> Affects Versions: 2.8.0
> Reporter: Karthik Kambatla
> Assignee: Szilard Nemeth
> Priority: Major
> Labels: newbie++
> Attachments: YARN-5106.001.patch
>
>
> Most, if not all, fair scheduler tests create an allocations XML file. Having a helper class that potentially uses a builder would make the tests cleaner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org