You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Oleksandr Shevchenko (JIRA)" <ji...@apache.org> on 2018/09/11 09:05:00 UTC
[jira] [Updated] (MAPREDUCE-7139)
TestShuffleProvider#testShuffleProviders and
TestTaskAttemptContainerRequest#testAttemptContainerRequest fail since a
static common container launch context does't recreate
[ https://issues.apache.org/jira/browse/MAPREDUCE-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Oleksandr Shevchenko updated MAPREDUCE-7139:
--------------------------------------------
Summary: TestShuffleProvider#testShuffleProviders and TestTaskAttemptContainerRequest#testAttemptContainerRequest fail since a static common container launch context does't recreate (was: TestShuffleProvider#testShuffleProviders and fails since a static common container launch context does't recreate)
> TestShuffleProvider#testShuffleProviders and TestTaskAttemptContainerRequest#testAttemptContainerRequest fail since a static common container launch context does't recreate
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7139
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7139
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Reporter: Oleksandr Shevchenko
> Assignee: Oleksandr Shevchenko
> Priority: Minor
> Attachments: MAPREDUCE-7139.001.patch, screenshot-1.png, screenshot-2.png
>
>
> TestShuffleProvider#testShuffleProviders fails with the following error:
> {noformat}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at org.apache.hadoop.mapreduce.v2.app.job.impl.TestShuffleProvider.testShuffleProviders(TestShuffleProvider.java:114)
> {noformat}
> This error occurs only when we run TestShuffleProvider#testShuffleProviders test after some test which creates TaskAttemptImpl (For example after tests in TestMapReduceChildJVM). !screenshot-1.png!
> TestShuffleProvider#testShuffleProviders creates container launch context, set additionals NM aux services and expects to get these services from container launch context after creating.
> But we have the following code in TaskAttemptImpl:
> {code}
> static ContainerLaunchContext createContainerLaunchContext(
> Map<ApplicationAccessType, String> applicationACLs,
> Configuration conf, Token<JobTokenIdentifier> jobToken, Task remoteTask,
> final org.apache.hadoop.mapred.JobID oldJobId,
> WrappedJvmID jvmID,
> TaskAttemptListener taskAttemptListener,
> Credentials credentials) {
> synchronized (commonContainerSpecLock) {
> if (commonContainerSpec == null) {
> commonContainerSpec = createCommonContainerLaunchContext(
> applicationACLs, conf, jobToken, oldJobId, credentials);
> }
> }
> {code}
> The property "commonContainerSpec" is static. As the result, we do not recreate common container launch within JVM. This is normal for AM JVM since all containers should be run with one context. But this leads to an error when we run unit tests in one JVM and expect to recreate container launch context.
> We can add a setter for "commonContainerSpec" with "VisibleForTesting" annotation and reset CLC in a test if we need it.
> Also, we can cleanup CLC after running tests in @After fixture which create CLC but we have too many tests which use TaskAttemptImpl and do not need cleanup.
> The same issue occurre in TestTaskAttemptContainerRequest#testAttemptContainerRequest. !screenshot-2.png! Since we create CLC by TaskAttemptImpl.createContainerLaunchContext.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org