You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karthik Kambatla (JIRA)" <ji...@apache.org> on 2014/09/18 19:14:34 UTC
[jira] [Comment Edited] (MAPREDUCE-6086) change
"mapreduce.job.credentials.binary" configuration to handle all URIs
properly.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138355#comment-14138355 ]
Karthik Kambatla edited comment on MAPREDUCE-6086 at 9/18/14 5:13 PM:
----------------------------------------------------------------------
I attached the IOException stacktrace for the TestBinaryTokenFile without my fix:
{noformat}
2014-09-17 18:08:42,490 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(446)) - Cleaning up the staging area /tmp/hadoop-yarn/staging/zxu/.staging/job_1411002502875_0002
Job failed with Exception reading file:/file:/Users/zxu/upstream/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/tokenFile.binary
java.io.IOException: Exception reading file:/file:/Users/zxu/upstream/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/tokenFile.binary
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:173)
at org.apache.hadoop.mapreduce.JobSubmitter.readTokensFromFiles(JobSubmitter.java:580)
at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:614)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:373)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1291)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1291)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1312)
at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.mapreduce.security.TestBinaryTokenFile.testTokenCacheFile(TestBinaryTokenFile.java:283)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.FileNotFoundException: File file:/file:/Users/zxu/upstream/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/tokenFile.binary does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:530)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:739)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:168)
... 38 more
{noformat}
was (Author: zxu):
I attached the IOException stacktrace for the TestBinaryTokenFile without my fix:
2014-09-17 18:08:42,490 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(446)) - Cleaning up the staging area /tmp/hadoop-yarn/staging/zxu/.staging/job_1411002502875_0002
Job failed with Exception reading file:/file:/Users/zxu/upstream/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/tokenFile.binary
java.io.IOException: Exception reading file:/file:/Users/zxu/upstream/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/tokenFile.binary
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:173)
at org.apache.hadoop.mapreduce.JobSubmitter.readTokensFromFiles(JobSubmitter.java:580)
at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:614)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:373)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1294)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1291)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1626)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1291)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1312)
at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.mapreduce.security.TestBinaryTokenFile.testTokenCacheFile(TestBinaryTokenFile.java:283)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.FileNotFoundException: File file:/file:/Users/zxu/upstream/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/test-dir/tokenFile.binary does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:530)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:739)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:168)
... 38 more
> change "mapreduce.job.credentials.binary" configuration to handle all URIs properly.
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6086
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6086
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: security
> Affects Versions: 2.5.0
> Reporter: zhihai xu
> Assignee: zhihai xu
> Attachments: MAPREDUCE-6086.000.patch
>
>
> Change "mapreduce.job.credentials.binary" configuration to handle all URIs properly.
> The current "mapreduce.job.credentials.binary" configuration only support local fs, It would be better to make it support non-local FS URIs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)