You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Yura Taras <yu...@gmail.com> on 2010/01/27 17:41:03 UTC
Failed to install Hadoop on WinXP
Hi all
I'm trying to deploy pseudo-distributed cluster on my devbox which
runs under WinXP. I did following steps:
1. Installed cygwin with ssh, configured ssh
2. Downloaded hadoop and extracted it, set JAVA_HOME and HADOOP_HOME
env vars (I made a symlink to java home, so it don't contain spaces)
3. Adjusted conf/hadoop-env.sh to point to correct JAVA_HOME
4. Adjusted conf files to following values:
* core-site.xml:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hdfs/hadoop</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8888</value>
</property>
</configuration>
* hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
* mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9999</value>
</property>
</configuration>
5. Next I execute following line: $ bin/hadoop namenode -format &&
bin/start-all.sh && bin/hadoop fs -put conf input && bin/hadoop jar
hadoop-0.20.1-examples.jar grep input output 'dfs[a-z.]'
I receive following exception:
localhost: starting tasktracker, logging to
/home/ytaras/hadoop/bin/../logs/hadoop-ytaras-tasktracker-bueno.out
10/01/27 18:23:55 INFO mapred.FileInputFormat: Total input paths to process : 13
10/01/27 18:23:56 INFO mapred.JobClient: Running job: job_201001271823_0001
10/01/27 18:23:57 INFO mapred.JobClient: map 0% reduce 0%
10/01/27 18:24:09 INFO mapred.JobClient: Task Id :
attempt_201001271823_0001_m_000014_0, Status : FAILED
java.io.FileNotFoundException: File
D:/hdfs/hadoop/mapred/local/taskTracker/jobcache/job_201001271823_0001/attempt_201001271823_0001_m_000014_0/work/tmp
does not exist.
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
!! SKIP - above exception few times !!
10/01/27 18:24:51 INFO mapred.JobClient: Job complete: job_201001271823_0001
10/01/27 18:24:51 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Am I doing something wrong (don't say just 'use Linux' :-) )?
Thanks
Re: Failed to install Hadoop on WinXP
Posted by Ed Mazur <ma...@cs.umass.edu>.
I tried running 0.20.0 on XP too a few weeks ago and stuck at the same
spot. No problems with standalone mode. Any insight would be
appreciated, thanks.
Ed
On Wed, Jan 27, 2010 at 11:41 AM, Yura Taras <yu...@gmail.com> wrote:
> Hi all
> I'm trying to deploy pseudo-distributed cluster on my devbox which
> runs under WinXP. I did following steps:
> 1. Installed cygwin with ssh, configured ssh
> 2. Downloaded hadoop and extracted it, set JAVA_HOME and HADOOP_HOME
> env vars (I made a symlink to java home, so it don't contain spaces)
> 3. Adjusted conf/hadoop-env.sh to point to correct JAVA_HOME
> 4. Adjusted conf files to following values:
> * core-site.xml:
> <configuration>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/hdfs/hadoop</value>
> <description>A base for other temporary directories.</description>
> </property>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://localhost:8888</value>
> </property>
> </configuration>
>
> * hdfs-site.xml:
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
> * mapred-site.xml:
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>localhost:9999</value>
> </property>
> </configuration>
>
> 5. Next I execute following line: $ bin/hadoop namenode -format &&
> bin/start-all.sh && bin/hadoop fs -put conf input && bin/hadoop jar
> hadoop-0.20.1-examples.jar grep input output 'dfs[a-z.]'
>
> I receive following exception:
> localhost: starting tasktracker, logging to
> /home/ytaras/hadoop/bin/../logs/hadoop-ytaras-tasktracker-bueno.out
> 10/01/27 18:23:55 INFO mapred.FileInputFormat: Total input paths to process : 13
> 10/01/27 18:23:56 INFO mapred.JobClient: Running job: job_201001271823_0001
> 10/01/27 18:23:57 INFO mapred.JobClient: map 0% reduce 0%
> 10/01/27 18:24:09 INFO mapred.JobClient: Task Id :
> attempt_201001271823_0001_m_000014_0, Status : FAILED
> java.io.FileNotFoundException: File
> D:/hdfs/hadoop/mapred/local/taskTracker/jobcache/job_201001271823_0001/attempt_201001271823_0001_m_000014_0/work/tmp
> does not exist.
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> !! SKIP - above exception few times !!
>
> 10/01/27 18:24:51 INFO mapred.JobClient: Job complete: job_201001271823_0001
> 10/01/27 18:24:51 INFO mapred.JobClient: Counters: 0
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.hadoop.examples.Grep.run(Grep.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.Grep.main(Grep.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
>
> Am I doing something wrong (don't say just 'use Linux' :-) )?
> Thanks
>
Re: Failed to install Hadoop on WinXP
Posted by brian <br...@gmail.com>.
Yura Taras wrote:
> Unfortunately, setting mapred.child.tmp doesn't help. Could you share
> your sample config files?
> What about VMWare - I am thinking about this as a last resort :)
>
> On Thu, Jan 28, 2010 at 3:41 PM, Yang Li <li...@cn.ibm.com> wrote:
>
>> I met the same problem on WinXP+Cygwin and fixed it by
>> either:
>> - moving to a linux box (VMWare works very well)
>> or:
>> - configuring a "mapred.child.tmp" parameter in core-site.xml
>>
>> I cannot explain why and how "mapred.child.tmp" is related to the problem.
>> From source code, it seems to be a JVM issue on windows. Hope this helps.
>>
>> Best regards,
>> -----------------------
>> Li Yang, ext. 22056, User Technologies Development, Shanghai, China
>>
>>
>>
>>
>> Yura Taras <yu...@gmail.com>
>> 2010-01-28 00:41
>> Please respond to
>> common-user@hadoop.apache.org
>>
>>
>> To
>> common-user@hadoop.apache.org
>> cc
>>
>> Subject
>> Failed to install Hadoop on WinXP
>>
>>
>>
>>
>>
>>
>> Hi all
>> I'm trying to deploy pseudo-distributed cluster on my devbox which
>> runs under WinXP. I did following steps:
>> 1. Installed cygwin with ssh, configured ssh
>> 2. Downloaded hadoop and extracted it, set JAVA_HOME and HADOOP_HOME
>> env vars (I made a symlink to java home, so it don't contain spaces)
>> 3. Adjusted conf/hadoop-env.sh to point to correct JAVA_HOME
>> 4. Adjusted conf files to following values:
>> * core-site.xml:
>> <configuration>
>> <property>
>> <name>hadoop.tmp.dir</name>
>> <value>/hdfs/hadoop</value>
>> <description>A base for other temporary directories.</description>
>> </property>
>> <property>
>> <name>fs.default.name</name>
>> <value>hdfs://localhost:8888</value>
>> </property>
>> </configuration>
>>
>> * hdfs-site.xml:
>> <configuration>
>> <property>
>> <name>dfs.replication</name>
>> <value>1</value>
>> </property>
>> </configuration>
>>
>> * mapred-site.xml:
>> <configuration>
>> <property>
>> <name>mapred.job.tracker</name>
>> <value>localhost:9999</value>
>> </property>
>> </configuration>
>>
>> 5. Next I execute following line: $ bin/hadoop namenode -format &&
>> bin/start-all.sh && bin/hadoop fs -put conf input && bin/hadoop jar
>> hadoop-0.20.1-examples.jar grep input output 'dfs[a-z.]'
>>
>> I receive following exception:
>> localhost: starting tasktracker, logging to
>> /home/ytaras/hadoop/bin/../logs/hadoop-ytaras-tasktracker-bueno.out
>> 10/01/27 18:23:55 INFO mapred.FileInputFormat: Total input paths to
>> process : 13
>> 10/01/27 18:23:56 INFO mapred.JobClient: Running job:
>> job_201001271823_0001
>> 10/01/27 18:23:57 INFO mapred.JobClient: map 0% reduce 0%
>> 10/01/27 18:24:09 INFO mapred.JobClient: Task Id :
>> attempt_201001271823_0001_m_000014_0, Status : FAILED
>> java.io.FileNotFoundException: File
>> D:/hdfs/hadoop/mapred/local/taskTracker/jobcache/job_201001271823_0001/attempt_201001271823_0001_m_000014_0/work/tmp
>> does not exist.
>> at
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
>> at
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
>> at
>> org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
>> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>>
>> !! SKIP - above exception few times !!
>>
>> 10/01/27 18:24:51 INFO mapred.JobClient: Job complete:
>> job_201001271823_0001
>> 10/01/27 18:24:51 INFO mapred.JobClient: Counters: 0
>> java.io.IOException: Job failed!
>> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>> at org.apache.hadoop.examples.Grep.run(Grep.java:69)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> at org.apache.hadoop.examples.Grep.main(Grep.java:93)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> at
>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> at
>> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>>
>>
>> Am I doing something wrong (don't say just 'use Linux' :-) )?
>> Thanks
>>
>>
>>
I've run into these problems too. I think you'll find this interesting:
Karmasphere Studio for Hadoop. http://www.hadoopstudio.org/
although I haven't fully tested it myself
Brian
Re: Failed to install Hadoop on WinXP
Posted by Yura Taras <yu...@gmail.com>.
Unfortunately, setting mapred.child.tmp doesn't help. Could you share
your sample config files?
What about VMWare - I am thinking about this as a last resort :)
On Thu, Jan 28, 2010 at 3:41 PM, Yang Li <li...@cn.ibm.com> wrote:
> I met the same problem on WinXP+Cygwin and fixed it by
> either:
> - moving to a linux box (VMWare works very well)
> or:
> - configuring a "mapred.child.tmp" parameter in core-site.xml
>
> I cannot explain why and how "mapred.child.tmp" is related to the problem.
> From source code, it seems to be a JVM issue on windows. Hope this helps.
>
> Best regards,
> -----------------------
> Li Yang, ext. 22056, User Technologies Development, Shanghai, China
>
>
>
>
> Yura Taras <yu...@gmail.com>
> 2010-01-28 00:41
> Please respond to
> common-user@hadoop.apache.org
>
>
> To
> common-user@hadoop.apache.org
> cc
>
> Subject
> Failed to install Hadoop on WinXP
>
>
>
>
>
>
> Hi all
> I'm trying to deploy pseudo-distributed cluster on my devbox which
> runs under WinXP. I did following steps:
> 1. Installed cygwin with ssh, configured ssh
> 2. Downloaded hadoop and extracted it, set JAVA_HOME and HADOOP_HOME
> env vars (I made a symlink to java home, so it don't contain spaces)
> 3. Adjusted conf/hadoop-env.sh to point to correct JAVA_HOME
> 4. Adjusted conf files to following values:
> * core-site.xml:
> <configuration>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/hdfs/hadoop</value>
> <description>A base for other temporary directories.</description>
> </property>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://localhost:8888</value>
> </property>
> </configuration>
>
> * hdfs-site.xml:
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> </configuration>
>
> * mapred-site.xml:
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>localhost:9999</value>
> </property>
> </configuration>
>
> 5. Next I execute following line: $ bin/hadoop namenode -format &&
> bin/start-all.sh && bin/hadoop fs -put conf input && bin/hadoop jar
> hadoop-0.20.1-examples.jar grep input output 'dfs[a-z.]'
>
> I receive following exception:
> localhost: starting tasktracker, logging to
> /home/ytaras/hadoop/bin/../logs/hadoop-ytaras-tasktracker-bueno.out
> 10/01/27 18:23:55 INFO mapred.FileInputFormat: Total input paths to
> process : 13
> 10/01/27 18:23:56 INFO mapred.JobClient: Running job:
> job_201001271823_0001
> 10/01/27 18:23:57 INFO mapred.JobClient: map 0% reduce 0%
> 10/01/27 18:24:09 INFO mapred.JobClient: Task Id :
> attempt_201001271823_0001_m_000014_0, Status : FAILED
> java.io.FileNotFoundException: File
> D:/hdfs/hadoop/mapred/local/taskTracker/jobcache/job_201001271823_0001/attempt_201001271823_0001_m_000014_0/work/tmp
> does not exist.
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
> at
> org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
> at org.apache.hadoop.mapred.Child.main(Child.java:155)
>
> !! SKIP - above exception few times !!
>
> 10/01/27 18:24:51 INFO mapred.JobClient: Job complete:
> job_201001271823_0001
> 10/01/27 18:24:51 INFO mapred.JobClient: Counters: 0
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
> at org.apache.hadoop.examples.Grep.run(Grep.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.Grep.main(Grep.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
>
> Am I doing something wrong (don't say just 'use Linux' :-) )?
> Thanks
>
>
RE: Failed to install Hadoop on WinXP
Posted by #YONG YONG CHENG# <aa...@pmail.ntu.edu.sg>.
Good Day,
My Hadoop version is 0.19.1. I have successfully configured to run it on a Windows machine.
Here is the configuration that I performed:
1. I put the Hadoop files under this folder C:\cygwin\usr\local\hadoop
2. Below is the hadoop-site.xml that I use.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/cygwin/tmp/hadoop-${user.name}</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
<property>
<name>mapred.job.reuse.jvm.num.tasks</name>
<value>-1</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>webinterface.private.actions</name>
<value>true</value>
</property>
</configuration>
-----Original Message-----
From: Yang Li [mailto:liyang@cn.ibm.com]
Sent: Thu 1/28/2010 9:41 PM
To: common-user@hadoop.apache.org
Subject: Re: Failed to install Hadoop on WinXP
I met the same problem on WinXP+Cygwin and fixed it by
either:
- moving to a linux box (VMWare works very well)
or:
- configuring a "mapred.child.tmp" parameter in core-site.xml
I cannot explain why and how "mapred.child.tmp" is related to the problem.
>From source code, it seems to be a JVM issue on windows. Hope this helps.
Best regards,
-----------------------
Li Yang, ext. 22056, User Technologies Development, Shanghai, China
Yura Taras <yu...@gmail.com>
2010-01-28 00:41
Please respond to
common-user@hadoop.apache.org
To
common-user@hadoop.apache.org
cc
Subject
Failed to install Hadoop on WinXP
Hi all
I'm trying to deploy pseudo-distributed cluster on my devbox which
runs under WinXP. I did following steps:
1. Installed cygwin with ssh, configured ssh
2. Downloaded hadoop and extracted it, set JAVA_HOME and HADOOP_HOME
env vars (I made a symlink to java home, so it don't contain spaces)
3. Adjusted conf/hadoop-env.sh to point to correct JAVA_HOME
4. Adjusted conf files to following values:
* core-site.xml:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hdfs/hadoop</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8888</value>
</property>
</configuration>
* hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
* mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9999</value>
</property>
</configuration>
5. Next I execute following line: $ bin/hadoop namenode -format &&
bin/start-all.sh && bin/hadoop fs -put conf input && bin/hadoop jar
hadoop-0.20.1-examples.jar grep input output 'dfs[a-z.]'
I receive following exception:
localhost: starting tasktracker, logging to
/home/ytaras/hadoop/bin/../logs/hadoop-ytaras-tasktracker-bueno.out
10/01/27 18:23:55 INFO mapred.FileInputFormat: Total input paths to
process : 13
10/01/27 18:23:56 INFO mapred.JobClient: Running job:
job_201001271823_0001
10/01/27 18:23:57 INFO mapred.JobClient: map 0% reduce 0%
10/01/27 18:24:09 INFO mapred.JobClient: Task Id :
attempt_201001271823_0001_m_000014_0, Status : FAILED
java.io.FileNotFoundException: File
D:/hdfs/hadoop/mapred/local/taskTracker/jobcache/job_201001271823_0001/attempt_201001271823_0001_m_000014_0/work/tmp
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at
org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
!! SKIP - above exception few times !!
10/01/27 18:24:51 INFO mapred.JobClient: Job complete:
job_201001271823_0001
10/01/27 18:24:51 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Am I doing something wrong (don't say just 'use Linux' :-) )?
Thanks
Re: Failed to install Hadoop on WinXP
Posted by Yang Li <li...@cn.ibm.com>.
I met the same problem on WinXP+Cygwin and fixed it by
either:
- moving to a linux box (VMWare works very well)
or:
- configuring a "mapred.child.tmp" parameter in core-site.xml
I cannot explain why and how "mapred.child.tmp" is related to the problem.
>From source code, it seems to be a JVM issue on windows. Hope this helps.
Best regards,
-----------------------
Li Yang, ext. 22056, User Technologies Development, Shanghai, China
Yura Taras <yu...@gmail.com>
2010-01-28 00:41
Please respond to
common-user@hadoop.apache.org
To
common-user@hadoop.apache.org
cc
Subject
Failed to install Hadoop on WinXP
Hi all
I'm trying to deploy pseudo-distributed cluster on my devbox which
runs under WinXP. I did following steps:
1. Installed cygwin with ssh, configured ssh
2. Downloaded hadoop and extracted it, set JAVA_HOME and HADOOP_HOME
env vars (I made a symlink to java home, so it don't contain spaces)
3. Adjusted conf/hadoop-env.sh to point to correct JAVA_HOME
4. Adjusted conf files to following values:
* core-site.xml:
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hdfs/hadoop</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:8888</value>
</property>
</configuration>
* hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
* mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9999</value>
</property>
</configuration>
5. Next I execute following line: $ bin/hadoop namenode -format &&
bin/start-all.sh && bin/hadoop fs -put conf input && bin/hadoop jar
hadoop-0.20.1-examples.jar grep input output 'dfs[a-z.]'
I receive following exception:
localhost: starting tasktracker, logging to
/home/ytaras/hadoop/bin/../logs/hadoop-ytaras-tasktracker-bueno.out
10/01/27 18:23:55 INFO mapred.FileInputFormat: Total input paths to
process : 13
10/01/27 18:23:56 INFO mapred.JobClient: Running job:
job_201001271823_0001
10/01/27 18:23:57 INFO mapred.JobClient: map 0% reduce 0%
10/01/27 18:24:09 INFO mapred.JobClient: Task Id :
attempt_201001271823_0001_m_000014_0, Status : FAILED
java.io.FileNotFoundException: File
D:/hdfs/hadoop/mapred/local/taskTracker/jobcache/job_201001271823_0001/attempt_201001271823_0001_m_000014_0/work/tmp
does not exist.
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
at
org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:519)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
!! SKIP - above exception few times !!
10/01/27 18:24:51 INFO mapred.JobClient: Job complete:
job_201001271823_0001
10/01/27 18:24:51 INFO mapred.JobClient: Counters: 0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.hadoop.examples.Grep.run(Grep.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.Grep.main(Grep.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Am I doing something wrong (don't say just 'use Linux' :-) )?
Thanks