You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ryan Lynch (JIRA)" <ji...@apache.org> on 2007/10/08 00:22:50 UTC
[jira] Created: (HADOOP-2007) Jobs use incorrect path to job.xml
for different users
Jobs use incorrect path to job.xml for different users
------------------------------------------------------
Key: HADOOP-2007
URL: https://issues.apache.org/jira/browse/HADOOP-2007
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.14.1
Environment: OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 5) - AMD64
JRE: java full version "1.6.0_02-b05"
Reporter: Ryan Lynch
We run hadoop/hdfs under a generic username of 'hadoop'. When submitting a job as user 'fred', the job.xml file gets created correctly as /tmp/*hadoop-fred*/mapred/system/<job_id>/job.xml, but the JobInProgress constructor appears to use the incorrect path of /tmp/*hadoop-hadoop*/mapred/system/<job_id>/job.xml for the copyToLocalFile operation. This results in an exception because the job.xml file does not exist under /tmp/hadoop-hadoop.
I think the incorrect path gets created on line 133 in JobInProgress.java right before the call to copyToLocalFile. If I hardcode the correct path (just to test my theory) of "/tmp/hadoop-fred/mapred/system" in place of the call to default_conf.getSystemDir(), the job kicks off and runs to completion as the user 'fred' with hadoop/hdfs running as the user 'hadoop'.
Here's a portion of the output when the job is submitted by user 'fred':
[fred@hdm01 ~]$ hadoop --config /usr/local/hadoop/conf jar /usr/local/hadoop/current/hadoop-0.14.1-examples.jar wordcount mytest/input/data output2
07/10/07 17:52:14 INFO mapred.FileInputFormat: Total input paths to process : 62
org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-hadoop/mapred/system/job_200710071751_0002/job.xml: No such file or directory
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:138)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:803)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:784)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:134)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1479)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-2007) Jobs use incorrect path to job.xml
for different users
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-2007.
-----------------------------------
Resolution: Won't Fix
> Jobs use incorrect path to job.xml for different users
> ------------------------------------------------------
>
> Key: HADOOP-2007
> URL: https://issues.apache.org/jira/browse/HADOOP-2007
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.1
> Environment: OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 5) - AMD64
> JRE: java full version "1.6.0_02-b05"
> Reporter: Ryan Lynch
>
> We run hadoop/hdfs under a generic username of 'hadoop'. When submitting a job as user 'fred', the job.xml file gets created correctly as /tmp/*hadoop-fred*/mapred/system/<job_id>/job.xml, but the JobInProgress constructor appears to use the incorrect path of /tmp/*hadoop-hadoop*/mapred/system/<job_id>/job.xml for the copyToLocalFile operation. This results in an exception because the job.xml file does not exist under /tmp/hadoop-hadoop.
> I think the incorrect path gets created on line 133 in JobInProgress.java right before the call to copyToLocalFile. If I hardcode the correct path (just to test my theory) of "/tmp/hadoop-fred/mapred/system" in place of the call to default_conf.getSystemDir(), the job kicks off and runs to completion as the user 'fred' with hadoop/hdfs running as the user 'hadoop'.
> Here's a portion of the output when the job is submitted by user 'fred':
> [fred@hdm01 ~]$ hadoop --config /usr/local/hadoop/conf jar /usr/local/hadoop/current/hadoop-0.14.1-examples.jar wordcount mytest/input/data output2
> 07/10/07 17:52:14 INFO mapred.FileInputFormat: Total input paths to process : 62
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-hadoop/mapred/system/job_200710071751_0002/job.xml: No such file or directory
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:138)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:803)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:784)
> at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:134)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1479)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2007) Jobs use incorrect path to job.xml
for different users
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533020 ]
Owen O'Malley commented on HADOOP-2007:
---------------------------------------
Please configure mapred.system.dir to be a constant and somewhere outside of /tmp. I personally like to use something like /hadoop/system so that it is very clear that this is a framework directory.
> Jobs use incorrect path to job.xml for different users
> ------------------------------------------------------
>
> Key: HADOOP-2007
> URL: https://issues.apache.org/jira/browse/HADOOP-2007
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.14.1
> Environment: OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 5) - AMD64
> JRE: java full version "1.6.0_02-b05"
> Reporter: Ryan Lynch
>
> We run hadoop/hdfs under a generic username of 'hadoop'. When submitting a job as user 'fred', the job.xml file gets created correctly as /tmp/*hadoop-fred*/mapred/system/<job_id>/job.xml, but the JobInProgress constructor appears to use the incorrect path of /tmp/*hadoop-hadoop*/mapred/system/<job_id>/job.xml for the copyToLocalFile operation. This results in an exception because the job.xml file does not exist under /tmp/hadoop-hadoop.
> I think the incorrect path gets created on line 133 in JobInProgress.java right before the call to copyToLocalFile. If I hardcode the correct path (just to test my theory) of "/tmp/hadoop-fred/mapred/system" in place of the call to default_conf.getSystemDir(), the job kicks off and runs to completion as the user 'fred' with hadoop/hdfs running as the user 'hadoop'.
> Here's a portion of the output when the job is submitted by user 'fred':
> [fred@hdm01 ~]$ hadoop --config /usr/local/hadoop/conf jar /usr/local/hadoop/current/hadoop-0.14.1-examples.jar wordcount mytest/input/data output2
> 07/10/07 17:52:14 INFO mapred.FileInputFormat: Total input paths to process : 62
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: /tmp/hadoop-hadoop/mapred/system/job_200710071751_0002/job.xml: No such file or directory
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:138)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:803)
> at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:784)
> at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:134)
> at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1479)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.