You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Luca Pireddu (JIRA)" <ji...@apache.org> on 2012/05/08 16:19:49 UTC
[jira] [Commented] (MAPREDUCE-4000) Create jobTokenPassword file
fail when run hadoop pipes locally
[ https://issues.apache.org/jira/browse/MAPREDUCE-4000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270482#comment-13270482 ]
Luca Pireddu commented on MAPREDUCE-4000:
-----------------------------------------
Stack trace, from Hadoop 1.0.2:
{{
12/05/08 16:11:32 INFO mapred.JobClient: Task Id : attempt_201205081608_0001_m_000000_2, Status : FAILED
java.io.FileNotFoundException: /u/pireddu/Projects/pydoop.git/examples/wordcount/test/jobTokenPassword (Permission denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:188)
at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:184)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:255)
at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:236)
at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:315)
at org.apache.hadoop.mapred.pipes.Application.writePasswordToLocalFile(Application.java:170)
at org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:106)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
}}
> Create jobTokenPassword file fail when run hadoop pipes locally
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-4000
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4000
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: pipes
> Affects Versions: 1.0.1, 1.0.2, 0.23.3
> Environment: Fedora release 16
> Linux localhost.localdomain 3.2.7-1.fc16.x86_64 #1 SMP Tue Feb 21 01:40:47 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
> Maven home: /usr/local/apache-maven-3.0.4
> Java version: 1.7.0_03, vendor: Oracle Corporation
> Java home: /usr/java/jdk1.7.0_03/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.2.7-1.fc16.x86_64", arch: "amd64", family: "unix"
> Reporter: Changming Sun
> Priority: Minor
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Create jobTokenPassword file fail when run hadoop pipes locally.
> I have put such settings in my job conf.
> {code:xml}
> <property>
> <name>mapreduce.jobtracker.address</name>
> <value>local</value>
> </property>
> <property>
> <name>mapreduce.framework.name</name>
> <value>local</value>
> </property>
> <property>
> <name>fs.defaultFS</name>
> <value>file:///</value>
> </property>
> {code}
> This job has one map task ,and one reduce task. It will fail when running PipesReducer, because "jobTokenPassword" file is created at current working directory,with permission 0400.
> {code:title=org.apache.hadoop.mapred.pipes.Application.java}
> private void writePasswordToLocalFile(String localPasswordFile,
> byte[] password, JobConf conf) throws IOException {
> FileSystem localFs = FileSystem.getLocal(conf);
> Path localPath = new Path(localPasswordFile);
> FSDataOutputStream out = FileSystem.create(localFs, localPath,
> new FsPermission("400"));
> out.write(password);
> out.close();
> }
> {code}
> So, it will fail at the second time.
> In such situation,the application cannot be initialized, but the PipesReducer.close() method will be called after that ,so there will be NullPointer Exception raise in close() method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira