You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arpit Gupta (JIRA)" <ji...@apache.org> on 2013/06/26 01:50:20 UTC
[jira] [Commented] (MAPREDUCE-5354) some job submission fail when
one of the configured local disks on jt is read only
[ https://issues.apache.org/jira/browse/MAPREDUCE-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693508#comment-13693508 ]
Arpit Gupta commented on MAPREDUCE-5354:
----------------------------------------
So some job submissions will go through and some will fail based on which ones end up using the read only disk. It would be better if jobtracker could detect this bad disk and ignore it for future use. Otherwise users will have remove this dir from the config and restart jobtracker and thus loose all the jobs in progress or if they have recovery enabled they will start from scratch.
> some job submission fail when one of the configured local disks on jt is read only
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5354
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5354
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Arpit Gupta
>
> Following exception is seen in the jt
> {code}
> 2013-06-25 06:58:26,988 INFO org.apache.hadoop.mapred.JobTracker: jobToken generated and stored with users keys in /mapred/system/job_201306250645_0002/jobToken
> 2013-06-25 06:58:26,997 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user cause:java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system)
> 2013-06-25 06:58:26,997 INFO org.apache.hadoop.ipc.Server: IPC Server handler 33 on 50300, call submitJob(job_201306250645_0002, hdfs://hostname:8020/user/hrt_qa/.staging/job_201306250645_0002, org.apache.hadoop.security.Credentials@34e77781) from 68.142.244.30:56631: error: java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system)
> java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)
> Caused by: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:193)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:260)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1248)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1229)
> at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:411)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
> ... 12 more
> 2013-06-25 06:58:27,079 INFO org.apache.hadoop.mapred.JobInitializationPoller: Passing to Initializer Job Id :job_2013062506
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira