You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bas Hickendorff <hi...@gmail.com> on 2012/04/03 15:28:06 UTC
How to find out what file Hadoop is looking for
Hello all,
My map-reduce operation on Hadoop (running on Debian) is correctly
starting and finding the input file. However, just after starting the
map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
it does not state what file it cannot find, or where it is looking.
Does someone now about what file error is? See below for the complete
error.
Since the java error is in the chmod() function (judging from the
stack in the output), I assume it is a problem with the rights, but
how do I know what rights to change if it gives me no path?
Thanks in advance,
Bas
The output of the job:
hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
/user/hadoopmachine/input /user/hadoopmachine/output
Warning: $HADOOP_HOME is deprecated.
12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
****hdfs://localhost:9000/user/hadoopmachine/input
12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
attempt_201204030722_0004_m_000002_0, Status : FAILED
Error initializing attempt_201204030722_0004_m_000002_0:
ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
at java.lang.Thread.run(Thread.java:636)
12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
Re: How to find out what file Hadoop is looking for
Posted by Bas Hickendorff <hi...@gmail.com>.
Hello,
Yes, that question on stackoverflow is mine as well. I first posted
there, but then realized that in this mailgroup there would probably
be more Hadoop knowledge.
And you where right! For some reason the userlog folder in the log
folder was owned by root instead of hadoopmachine. I have no idea how
that happened, but I changed it and now it is running without errors.
Everybody thanks a lot for you time and help, much appreciated!
Chris: I will accept your answer on stackoverflow.
Regards,
Bas
On Tue, Apr 3, 2012 at 8:57 PM, Chris White <ch...@gmail.com> wrote:
> This looks like a log dir problem:
>
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>
> looking through the source for JobLocalizer, it's trying to create a
> folder under ${hadoop.log.dir}/userlogs. There's a similar question (i
> assume it's yours) on StackOverflow:
>
> http://stackoverflow.com/questions/9992566/hadoop-map-reduce-operation-is-failing-on-writing-output
>
> Basically from your ps trace, the hadoop.log.dir is pointing to
> /home/hadoopmachine/hadoop-1.0.1/libexec/../logs - check this folder
> is writable by your 'hadoopmachine' user:
>
> 1000 4249 2.2 0.8 1181992 30176 ? Sl 12:09 0:00
> /usr/lib/jvm/java-6-openjdk/bin/java -Dproc_tasktracker -Xmx1000m
> -Dhadoop.log.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/../logs
> -Dhadoop.log.file=hadoop-hadoopmachine-tasktracker-debian.log
> -Dhadoop.home.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/..
> -Dhadoop.id.str=hadoopmachine -Dhadoop.root.logger=INFO,DRFA
> -Dhadoop.security.logger=INFO,NullAppender
> -Djava.library.path=/home/hadoopmachine/hadoop-1.0.1/libexec/../lib/native/Linux-i386-32
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
>
>
>
> On Tue, Apr 3, 2012 at 2:34 PM, Bas Hickendorff
> <hi...@gmail.com> wrote:
>> ps -ef | grep hadoop shows that it is indeed "hadoopmachine" that is
>> running hadoop.
>>
>> I su'ed into the user hadoopmachine (which is also the standard user I
>> login with in debian), and I can access the hdfs that way as well.
>>
>> The free space should also not be a problem:
>>
>> hadoopmachine@debian:~$ df -h
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda1 7.6G 4.0G 3.2G 56% /
>> tmpfs 1.8G 0 1.8G 0% /lib/init/rw
>> udev 1.8G 172K 1.8G 1% /dev
>> tmpfs 1.8G 0 1.8G 0% /dev/shm
>>
>>
>> I don't know if it is relevant, but it is on a virtual machine.
>>
>>
>> Regards,
>>
>> Bas
>>
>>
>> On Tue, Apr 3, 2012 at 8:17 PM, Harsh J <ha...@cloudera.com> wrote:
>>> The permissions look alright if TT too is run by 'hadoopmachine'. Can
>>> you also check if you have adequate space free, reported by df -h
>>> /home/hadoopmachine?
>>>
>>> On Tue, Apr 3, 2012 at 10:28 PM, Bas Hickendorff
>>> <hi...@gmail.com> wrote:
>>>> Thanks for your help!
>>>> However, as far as I can see, the user has those rights.
>>>>
>>>> I have in mapred-ste.xml :
>>>>
>>>> <property>
>>>> <name>mapred.local.dir</name>
>>>> <value>/home/hadoopmachine/hadoop_data/mapred</value>
>>>> <final>true</final>
>>>> </property>
>>>>
>>>>
>>>> and the directories look like this:
>>>>
>>>> hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
>>>> hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
>>>> total 24K
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
>>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
>>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
>>>> drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
>>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
>>>>
>>>> hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
>>>> hadoopmachine@debian:~/hadoop_data$ ls -lah
>>>> total 24K
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
>>>> drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
>>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
>>>> drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
>>>> drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
>>>>
>>>>
>>>> As far as I can see (but my linux permissions knowledge might be
>>>> failing) the user "hadoopmachine" has rights on these folders. I
>>>> confirmed that that user is indeed the user that runs the TaskTracker.
>>>>
>>>> Are there any other things I could check?
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Bas
>>>>
>>>> On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
>>>>> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
>>>>> permissions set on them. Make sure they are owned by the user that
>>>>> runs the TT service and have read/write permission at least for that
>>>>> user.
>>>>>
>>>>> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
>>>>> <hi...@gmail.com> wrote:
>>>>>> Hello all,
>>>>>>
>>>>>> My map-reduce operation on Hadoop (running on Debian) is correctly
>>>>>> starting and finding the input file. However, just after starting the
>>>>>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>>>>> it does not state what file it cannot find, or where it is looking.
>>>>>> Does someone now about what file error is? See below for the complete
>>>>>> error.
>>>>>>
>>>>>> Since the java error is in the chmod() function (judging from the
>>>>>> stack in the output), I assume it is a problem with the rights, but
>>>>>> how do I know what rights to change if it gives me no path?
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Bas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The output of the job:
>>>>>>
>>>>>>
>>>>>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>>>>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>>>>> /user/hadoopmachine/input /user/hadoopmachine/output
>>>>>> Warning: $HADOOP_HOME is deprecated.
>>>>>>
>>>>>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>>> parsing the arguments. Applications should implement Tool for the
>>>>>> same.
>>>>>> ****hdfs://localhost:9000/user/hadoopmachine/input
>>>>>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
>>>>>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
>>>>>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>>>>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>>>>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>>>>>> Error initializing attempt_201204030722_0004_m_000002_0:
>>>>>> ENOENT: No such file or directory
>>>>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>>>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>>>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>>>>> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>>>>>> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>>>>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>>>> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>>>>>> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>>>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>>>>> at java.lang.Thread.run(Thread.java:636)
>>>>>>
>>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>
>>>
>>>
>>> --
>>> Harsh J
Re: How to find out what file Hadoop is looking for
Posted by Chris White <ch...@gmail.com>.
This looks like a log dir problem:
at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
looking through the source for JobLocalizer, it's trying to create a
folder under ${hadoop.log.dir}/userlogs. There's a similar question (i
assume it's yours) on StackOverflow:
http://stackoverflow.com/questions/9992566/hadoop-map-reduce-operation-is-failing-on-writing-output
Basically from your ps trace, the hadoop.log.dir is pointing to
/home/hadoopmachine/hadoop-1.0.1/libexec/../logs - check this folder
is writable by your 'hadoopmachine' user:
1000 4249 2.2 0.8 1181992 30176 ? Sl 12:09 0:00
/usr/lib/jvm/java-6-openjdk/bin/java -Dproc_tasktracker -Xmx1000m
-Dhadoop.log.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/../logs
-Dhadoop.log.file=hadoop-hadoopmachine-tasktracker-debian.log
-Dhadoop.home.dir=/home/hadoopmachine/hadoop-1.0.1/libexec/..
-Dhadoop.id.str=hadoopmachine -Dhadoop.root.logger=INFO,DRFA
-Dhadoop.security.logger=INFO,NullAppender
-Djava.library.path=/home/hadoopmachine/hadoop-1.0.1/libexec/../lib/native/Linux-i386-32
-Dhadoop.policy.file=hadoop-policy.xml -classpath
On Tue, Apr 3, 2012 at 2:34 PM, Bas Hickendorff
<hi...@gmail.com> wrote:
> ps -ef | grep hadoop shows that it is indeed "hadoopmachine" that is
> running hadoop.
>
> I su'ed into the user hadoopmachine (which is also the standard user I
> login with in debian), and I can access the hdfs that way as well.
>
> The free space should also not be a problem:
>
> hadoopmachine@debian:~$ df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda1 7.6G 4.0G 3.2G 56% /
> tmpfs 1.8G 0 1.8G 0% /lib/init/rw
> udev 1.8G 172K 1.8G 1% /dev
> tmpfs 1.8G 0 1.8G 0% /dev/shm
>
>
> I don't know if it is relevant, but it is on a virtual machine.
>
>
> Regards,
>
> Bas
>
>
> On Tue, Apr 3, 2012 at 8:17 PM, Harsh J <ha...@cloudera.com> wrote:
>> The permissions look alright if TT too is run by 'hadoopmachine'. Can
>> you also check if you have adequate space free, reported by df -h
>> /home/hadoopmachine?
>>
>> On Tue, Apr 3, 2012 at 10:28 PM, Bas Hickendorff
>> <hi...@gmail.com> wrote:
>>> Thanks for your help!
>>> However, as far as I can see, the user has those rights.
>>>
>>> I have in mapred-ste.xml :
>>>
>>> <property>
>>> <name>mapred.local.dir</name>
>>> <value>/home/hadoopmachine/hadoop_data/mapred</value>
>>> <final>true</final>
>>> </property>
>>>
>>>
>>> and the directories look like this:
>>>
>>> hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
>>> hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
>>> total 24K
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
>>> drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
>>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
>>>
>>> hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
>>> hadoopmachine@debian:~/hadoop_data$ ls -lah
>>> total 24K
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
>>> drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
>>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
>>> drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
>>> drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
>>>
>>>
>>> As far as I can see (but my linux permissions knowledge might be
>>> failing) the user "hadoopmachine" has rights on these folders. I
>>> confirmed that that user is indeed the user that runs the TaskTracker.
>>>
>>> Are there any other things I could check?
>>>
>>>
>>> Regards,
>>>
>>> Bas
>>>
>>> On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
>>>> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
>>>> permissions set on them. Make sure they are owned by the user that
>>>> runs the TT service and have read/write permission at least for that
>>>> user.
>>>>
>>>> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
>>>> <hi...@gmail.com> wrote:
>>>>> Hello all,
>>>>>
>>>>> My map-reduce operation on Hadoop (running on Debian) is correctly
>>>>> starting and finding the input file. However, just after starting the
>>>>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>>>> it does not state what file it cannot find, or where it is looking.
>>>>> Does someone now about what file error is? See below for the complete
>>>>> error.
>>>>>
>>>>> Since the java error is in the chmod() function (judging from the
>>>>> stack in the output), I assume it is a problem with the rights, but
>>>>> how do I know what rights to change if it gives me no path?
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Bas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The output of the job:
>>>>>
>>>>>
>>>>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>>>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>>>> /user/hadoopmachine/input /user/hadoopmachine/output
>>>>> Warning: $HADOOP_HOME is deprecated.
>>>>>
>>>>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>> parsing the arguments. Applications should implement Tool for the
>>>>> same.
>>>>> ****hdfs://localhost:9000/user/hadoopmachine/input
>>>>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
>>>>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
>>>>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>>>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>>>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>>>>> Error initializing attempt_201204030722_0004_m_000002_0:
>>>>> ENOENT: No such file or directory
>>>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>>>> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>>>> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>>>>> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>>>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>>> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>>>>> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>>>> at java.lang.Thread.run(Thread.java:636)
>>>>>
>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>>>>
>>>>
>>>>
>>>> --
>>>> Harsh J
>>
>>
>>
>> --
>> Harsh J
Re: How to find out what file Hadoop is looking for
Posted by Bas Hickendorff <hi...@gmail.com>.
ps -ef | grep hadoop shows that it is indeed "hadoopmachine" that is
running hadoop.
I su'ed into the user hadoopmachine (which is also the standard user I
login with in debian), and I can access the hdfs that way as well.
The free space should also not be a problem:
hadoopmachine@debian:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 7.6G 4.0G 3.2G 56% /
tmpfs 1.8G 0 1.8G 0% /lib/init/rw
udev 1.8G 172K 1.8G 1% /dev
tmpfs 1.8G 0 1.8G 0% /dev/shm
I don't know if it is relevant, but it is on a virtual machine.
Regards,
Bas
On Tue, Apr 3, 2012 at 8:17 PM, Harsh J <ha...@cloudera.com> wrote:
> The permissions look alright if TT too is run by 'hadoopmachine'. Can
> you also check if you have adequate space free, reported by df -h
> /home/hadoopmachine?
>
> On Tue, Apr 3, 2012 at 10:28 PM, Bas Hickendorff
> <hi...@gmail.com> wrote:
>> Thanks for your help!
>> However, as far as I can see, the user has those rights.
>>
>> I have in mapred-ste.xml :
>>
>> <property>
>> <name>mapred.local.dir</name>
>> <value>/home/hadoopmachine/hadoop_data/mapred</value>
>> <final>true</final>
>> </property>
>>
>>
>> and the directories look like this:
>>
>> hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
>> hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
>> total 24K
>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
>> drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
>> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
>>
>> hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
>> hadoopmachine@debian:~/hadoop_data$ ls -lah
>> total 24K
>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
>> drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
>> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
>> drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
>> drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
>>
>>
>> As far as I can see (but my linux permissions knowledge might be
>> failing) the user "hadoopmachine" has rights on these folders. I
>> confirmed that that user is indeed the user that runs the TaskTracker.
>>
>> Are there any other things I could check?
>>
>>
>> Regards,
>>
>> Bas
>>
>> On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
>>> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
>>> permissions set on them. Make sure they are owned by the user that
>>> runs the TT service and have read/write permission at least for that
>>> user.
>>>
>>> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
>>> <hi...@gmail.com> wrote:
>>>> Hello all,
>>>>
>>>> My map-reduce operation on Hadoop (running on Debian) is correctly
>>>> starting and finding the input file. However, just after starting the
>>>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>>> it does not state what file it cannot find, or where it is looking.
>>>> Does someone now about what file error is? See below for the complete
>>>> error.
>>>>
>>>> Since the java error is in the chmod() function (judging from the
>>>> stack in the output), I assume it is a problem with the rights, but
>>>> how do I know what rights to change if it gives me no path?
>>>>
>>>> Thanks in advance,
>>>>
>>>> Bas
>>>>
>>>>
>>>>
>>>>
>>>> The output of the job:
>>>>
>>>>
>>>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>>> /user/hadoopmachine/input /user/hadoopmachine/output
>>>> Warning: $HADOOP_HOME is deprecated.
>>>>
>>>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>>> parsing the arguments. Applications should implement Tool for the
>>>> same.
>>>> ****hdfs://localhost:9000/user/hadoopmachine/input
>>>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
>>>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
>>>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>>>> Error initializing attempt_201204030722_0004_m_000002_0:
>>>> ENOENT: No such file or directory
>>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>>> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>>> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>>>> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>>> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>>>> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>>> at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>>>
>>>
>>>
>>> --
>>> Harsh J
>
>
>
> --
> Harsh J
Re: How to find out what file Hadoop is looking for
Posted by Harsh J <ha...@cloudera.com>.
The permissions look alright if TT too is run by 'hadoopmachine'. Can
you also check if you have adequate space free, reported by df -h
/home/hadoopmachine?
On Tue, Apr 3, 2012 at 10:28 PM, Bas Hickendorff
<hi...@gmail.com> wrote:
> Thanks for your help!
> However, as far as I can see, the user has those rights.
>
> I have in mapred-ste.xml :
>
> <property>
> <name>mapred.local.dir</name>
> <value>/home/hadoopmachine/hadoop_data/mapred</value>
> <final>true</final>
> </property>
>
>
> and the directories look like this:
>
> hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
> hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
> total 24K
> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
> drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
> drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
>
> hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
> hadoopmachine@debian:~/hadoop_data$ ls -lah
> total 24K
> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
> drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
> drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
> drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
> drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
>
>
> As far as I can see (but my linux permissions knowledge might be
> failing) the user "hadoopmachine" has rights on these folders. I
> confirmed that that user is indeed the user that runs the TaskTracker.
>
> Are there any other things I could check?
>
>
> Regards,
>
> Bas
>
> On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
>> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
>> permissions set on them. Make sure they are owned by the user that
>> runs the TT service and have read/write permission at least for that
>> user.
>>
>> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
>> <hi...@gmail.com> wrote:
>>> Hello all,
>>>
>>> My map-reduce operation on Hadoop (running on Debian) is correctly
>>> starting and finding the input file. However, just after starting the
>>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>> it does not state what file it cannot find, or where it is looking.
>>> Does someone now about what file error is? See below for the complete
>>> error.
>>>
>>> Since the java error is in the chmod() function (judging from the
>>> stack in the output), I assume it is a problem with the rights, but
>>> how do I know what rights to change if it gives me no path?
>>>
>>> Thanks in advance,
>>>
>>> Bas
>>>
>>>
>>>
>>>
>>> The output of the job:
>>>
>>>
>>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>> /user/hadoopmachine/input /user/hadoopmachine/output
>>> Warning: $HADOOP_HOME is deprecated.
>>>
>>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>> parsing the arguments. Applications should implement Tool for the
>>> same.
>>> ****hdfs://localhost:9000/user/hadoopmachine/input
>>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
>>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
>>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>>> Error initializing attempt_201204030722_0004_m_000002_0:
>>> ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>>> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>>> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>>> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>>> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>>
>>
>>
>> --
>> Harsh J
--
Harsh J
Re: How to find out what file Hadoop is looking for
Posted by Bas Hickendorff <hi...@gmail.com>.
Thanks for your help!
However, as far as I can see, the user has those rights.
I have in mapred-ste.xml :
<property>
<name>mapred.local.dir</name>
<value>/home/hadoopmachine/hadoop_data/mapred</value>
<final>true</final>
</property>
and the directories look like this:
hadoopmachine@debian:~$ cd /home/hadoopmachine/hadoop_data/mapred
hadoopmachine@debian:~/hadoop_data/mapred$ ls -lah
total 24K
drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 .
drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 ..
drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 taskTracker
drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 tt_log_tmp
drwx------ 2 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 ttprivate
drwxr-xr-x 2 hadoopmachine hadoopmachine 4.0K Apr 3 08:28 userlogs
hadoopmachine@debian:~/hadoop_data/mapred$ cd ..
hadoopmachine@debian:~/hadoop_data$ ls -lah
total 24K
drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 08:26 .
drwxr-xr-x 31 hadoopmachine hadoopmachine 4.0K Apr 3 12:08 ..
drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:10 data
drwxr-xr-x 6 hadoopmachine hadoopmachine 4.0K Apr 3 12:11 mapred
drwxr-xr-x 5 hadoopmachine hadoopmachine 4.0K Apr 3 12:09 name
drwxr-xr-x 4 hadoopmachine hadoopmachine 4.0K Apr 3 10:11 tmp
As far as I can see (but my linux permissions knowledge might be
failing) the user "hadoopmachine" has rights on these folders. I
confirmed that that user is indeed the user that runs the TaskTracker.
Are there any other things I could check?
Regards,
Bas
On Tue, Apr 3, 2012 at 6:12 PM, Harsh J <ha...@cloudera.com> wrote:
> Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
> permissions set on them. Make sure they are owned by the user that
> runs the TT service and have read/write permission at least for that
> user.
>
> On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
> <hi...@gmail.com> wrote:
>> Hello all,
>>
>> My map-reduce operation on Hadoop (running on Debian) is correctly
>> starting and finding the input file. However, just after starting the
>> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>> it does not state what file it cannot find, or where it is looking.
>> Does someone now about what file error is? See below for the complete
>> error.
>>
>> Since the java error is in the chmod() function (judging from the
>> stack in the output), I assume it is a problem with the rights, but
>> how do I know what rights to change if it gives me no path?
>>
>> Thanks in advance,
>>
>> Bas
>>
>>
>>
>>
>> The output of the job:
>>
>>
>> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>> /user/hadoopmachine/input /user/hadoopmachine/output
>> Warning: $HADOOP_HOME is deprecated.
>>
>> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the
>> same.
>> ****hdfs://localhost:9000/user/hadoopmachine/input
>> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
>> 12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
>> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>> attempt_201204030722_0004_m_000002_0, Status : FAILED
>> Error initializing attempt_201204030722_0004_m_000002_0:
>> ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
>> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
>> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
>> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>> at java.lang.Thread.run(Thread.java:636)
>>
>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
>> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
>
>
>
> --
> Harsh J
Re: How to find out what file Hadoop is looking for
Posted by Harsh J <ha...@cloudera.com>.
Some of your TaskTrackers' mapred.local.dirs do not have proper r/w
permissions set on them. Make sure they are owned by the user that
runs the TT service and have read/write permission at least for that
user.
On Tue, Apr 3, 2012 at 6:58 PM, Bas Hickendorff
<hi...@gmail.com> wrote:
> Hello all,
>
> My map-reduce operation on Hadoop (running on Debian) is correctly
> starting and finding the input file. However, just after starting the
> map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
> it does not state what file it cannot find, or where it is looking.
> Does someone now about what file error is? See below for the complete
> error.
>
> Since the java error is in the chmod() function (judging from the
> stack in the output), I assume it is a problem with the rights, but
> how do I know what rights to change if it gives me no path?
>
> Thanks in advance,
>
> Bas
>
>
>
>
> The output of the job:
>
>
> hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
> hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
> /user/hadoopmachine/input /user/hadoopmachine/output
> Warning: $HADOOP_HOME is deprecated.
>
> 12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> ****hdfs://localhost:9000/user/hadoopmachine/input
> 12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to process : 1
> 12/04/03 08:05:08 INFO mapred.JobClient: Running job: job_201204030722_0004
> 12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
> 12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
> attempt_201204030722_0004_m_000002_0, Status : FAILED
> Error initializing attempt_201204030722_0004_m_000002_0:
> ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
> at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
> at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
> at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:239)
> at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:196)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
> at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
> at java.lang.Thread.run(Thread.java:636)
>
> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stdout
> 12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_201204030722_0004_m_000002_0&filter=stderr
--
Harsh J
Re: How to find out what file Hadoop is looking for
Posted by Serge Blazhievsky <Se...@nice.com>.
Take a look which user is actually running hadoop
Ps -ef | grep hadoop
Su to that user and try to touch hdfs directory
Regards
Serge
On 4/3/12 10:44 AM, "Bas Hickendorff" <hi...@gmail.com> wrote:
>Yes, it does, and it contains the file with input data (file is called
>"in").
>
>
>hadoopmachine@debian:~/hadoop-1.0.1$ bin/hadoop fs -ls
>Warning: $HADOOP_HOME is deprecated.
>
>Found 1 items
>drwxr-xr-x - hadoopmachine supergroup 0 2012-04-03 07:11
>/user/hadoopmachine/input
>
>
>hadoopmachine@debian:~/hadoop-1.0.1$ bin/hadoop fs -ls input
>Warning: $HADOOP_HOME is deprecated.
>
>Found 1 items
>-rw-r--r-- 1 hadoopmachine supergroup 74 2012-04-02 10:08
>/user/hadoopmachine/input/in
>
>
>Is this correct?
>
>Regards,
>
>Bas
>
>
>On Tue, Apr 3, 2012 at 7:12 PM, Serge Blazhievsky
><Se...@nice.com> wrote:
>> Does this directory exist in HDFS
>>
>>
>> /user/hadoopmachine/input
>>
>> ???
>>
>>
>> Serge Blazhievsky
>>
>>
>> On 4/3/12 6:28 AM, "Bas Hickendorff" <hi...@gmail.com> wrote:
>>
>>>Hello all,
>>>
>>>My map-reduce operation on Hadoop (running on Debian) is correctly
>>>starting and finding the input file. However, just after starting the
>>>map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>>it does not state what file it cannot find, or where it is looking.
>>>Does someone now about what file error is? See below for the complete
>>>error.
>>>
>>>Since the java error is in the chmod() function (judging from the
>>>stack in the output), I assume it is a problem with the rights, but
>>>how do I know what rights to change if it gives me no path?
>>>
>>>Thanks in advance,
>>>
>>>Bas
>>>
>>>
>>>
>>>
>>>The output of the job:
>>>
>>>
>>>hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>>hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>>/user/hadoopmachine/input /user/hadoopmachine/output
>>>Warning: $HADOOP_HOME is deprecated.
>>>
>>>12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>>parsing the arguments. Applications should implement Tool for the
>>>same.
>>>****hdfs://localhost:9000/user/hadoopmachine/input
>>>12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to
>>>process : 1
>>>12/04/03 08:05:08 INFO mapred.JobClient: Running job:
>>>job_201204030722_0004
>>>12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>>12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>>attempt_201204030722_0004_m_000002_0, Status : FAILED
>>>Error initializing attempt_201204030722_0004_m_000002_0:
>>>ENOENT: No such file or directory
>>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>> at
>>>org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>>> at
>>>org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem
>>>.j
>>>ava:509)
>>> at
>>>org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:3
>>>44
>>>)
>>> at
>>>org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.j
>>>av
>>>a:239)
>>> at
>>>org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTask
>>>Co
>>>ntroller.java:196)
>>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>> at
>>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
>>>n.
>>>java:1093)
>>> at
>>>org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201
>>>)
>>> at
>>>org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>>> at java.lang.Thread.run(Thread.java:636)
>>>
>>>12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_20
>>>12
>>>04030722_0004_m_000002_0&filter=stdout
>>>12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>>outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_20
>>>12
>>>04030722_0004_m_000002_0&filter=stderr
>>
Re: How to find out what file Hadoop is looking for
Posted by Bas Hickendorff <hi...@gmail.com>.
Yes, it does, and it contains the file with input data (file is called "in").
hadoopmachine@debian:~/hadoop-1.0.1$ bin/hadoop fs -ls
Warning: $HADOOP_HOME is deprecated.
Found 1 items
drwxr-xr-x - hadoopmachine supergroup 0 2012-04-03 07:11
/user/hadoopmachine/input
hadoopmachine@debian:~/hadoop-1.0.1$ bin/hadoop fs -ls input
Warning: $HADOOP_HOME is deprecated.
Found 1 items
-rw-r--r-- 1 hadoopmachine supergroup 74 2012-04-02 10:08
/user/hadoopmachine/input/in
Is this correct?
Regards,
Bas
On Tue, Apr 3, 2012 at 7:12 PM, Serge Blazhievsky
<Se...@nice.com> wrote:
> Does this directory exist in HDFS
>
>
> /user/hadoopmachine/input
>
> ???
>
>
> Serge Blazhievsky
>
>
> On 4/3/12 6:28 AM, "Bas Hickendorff" <hi...@gmail.com> wrote:
>
>>Hello all,
>>
>>My map-reduce operation on Hadoop (running on Debian) is correctly
>>starting and finding the input file. However, just after starting the
>>map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>>it does not state what file it cannot find, or where it is looking.
>>Does someone now about what file error is? See below for the complete
>>error.
>>
>>Since the java error is in the chmod() function (judging from the
>>stack in the output), I assume it is a problem with the rights, but
>>how do I know what rights to change if it gives me no path?
>>
>>Thanks in advance,
>>
>>Bas
>>
>>
>>
>>
>>The output of the job:
>>
>>
>>hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>>hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>>/user/hadoopmachine/input /user/hadoopmachine/output
>>Warning: $HADOOP_HOME is deprecated.
>>
>>12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>>parsing the arguments. Applications should implement Tool for the
>>same.
>>****hdfs://localhost:9000/user/hadoopmachine/input
>>12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to
>>process : 1
>>12/04/03 08:05:08 INFO mapred.JobClient: Running job:
>>job_201204030722_0004
>>12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>>12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>>attempt_201204030722_0004_m_000002_0, Status : FAILED
>>Error initializing attempt_201204030722_0004_m_000002_0:
>>ENOENT: No such file or directory
>> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
>> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
>> at
>>org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.j
>>ava:509)
>> at
>>org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344
>>)
>> at
>>org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.jav
>>a:239)
>> at
>>org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskCo
>>ntroller.java:196)
>> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>> at
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>java:1093)
>> at
>>org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
>> at
>>org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
>> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
>> at java.lang.Thread.run(Thread.java:636)
>>
>>12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_2012
>>04030722_0004_m_000002_0&filter=stdout
>>12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>>outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_2012
>>04030722_0004_m_000002_0&filter=stderr
>
Re: How to find out what file Hadoop is looking for
Posted by Serge Blazhievsky <Se...@nice.com>.
Does this directory exist in HDFS
/user/hadoopmachine/input
???
Serge Blazhievsky
On 4/3/12 6:28 AM, "Bas Hickendorff" <hi...@gmail.com> wrote:
>Hello all,
>
>My map-reduce operation on Hadoop (running on Debian) is correctly
>starting and finding the input file. However, just after starting the
>map reduce, Hadoop tells me that it cannot find a file. Unfortunately,
>it does not state what file it cannot find, or where it is looking.
>Does someone now about what file error is? See below for the complete
>error.
>
>Since the java error is in the chmod() function (judging from the
>stack in the output), I assume it is a problem with the rights, but
>how do I know what rights to change if it gives me no path?
>
>Thanks in advance,
>
>Bas
>
>
>
>
>The output of the job:
>
>
>hadoopmachine@debian:~$ ./hadoop-1.0.1/bin/hadoop jar
>hadooptest/main.jar nl.mydomain.hadoop.debian.test.Main
>/user/hadoopmachine/input /user/hadoopmachine/output
>Warning: $HADOOP_HOME is deprecated.
>
>12/04/03 08:05:08 WARN mapred.JobClient: Use GenericOptionsParser for
>parsing the arguments. Applications should implement Tool for the
>same.
>****hdfs://localhost:9000/user/hadoopmachine/input
>12/04/03 08:05:08 INFO input.FileInputFormat: Total input paths to
>process : 1
>12/04/03 08:05:08 INFO mapred.JobClient: Running job:
>job_201204030722_0004
>12/04/03 08:05:09 INFO mapred.JobClient: map 0% reduce 0%
>12/04/03 08:05:13 INFO mapred.JobClient: Task Id :
>attempt_201204030722_0004_m_000002_0, Status : FAILED
>Error initializing attempt_201204030722_0004_m_000002_0:
>ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:692)
> at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:647)
> at
>org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.j
>ava:509)
> at
>org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344
>)
> at
>org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.jav
>a:239)
> at
>org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskCo
>ntroller.java:196)
> at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1093)
> at
>org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1201)
> at
>org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1116)
> at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2404)
> at java.lang.Thread.run(Thread.java:636)
>
>12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_2012
>04030722_0004_m_000002_0&filter=stdout
>12/04/03 08:05:13 WARN mapred.JobClient: Error reading task
>outputhttp://localhost:50060/tasklog?plaintext=true&attemptid=attempt_2012
>04030722_0004_m_000002_0&filter=stderr