You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Christopher Worley <ch...@gmail.com> on 2010/11/11 18:36:45 UTC
Hadoop installation on Windows
Hello,
I'm trying to get Hadoop working on Cygwin/Windows XP in
Pseudo-Distributed Mode. I downloaded version 0.21.0 and unpacked it
to a cygwin directory. I'm following the quickstart directions here:
http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
I get the following error when I try to check the Hadoop version:
$ bin/hadoop --version
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/util/PlatformName
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
Could not find the main class: org.apache.hadoop.util.PlatformName. Program wil
l exit.
Unrecognized option: --version
Could not create the Java virtual machine.
I get similar exceptions when I run "start-all.sh", but I am still
able to browse the web interface for NameNode and JobTracker.
I'm sure has something to do with an installation step that I'm
missing--some step that windows users have to do, that linux users do
not. I did web searches for this problem, but I only found it
occurring when people were building Hadoop. I'm not building.
I appreciate any help anyone can give.
Thanks,
Chris
Re: Hadoop installation on Windows
Posted by Steve Loughran <st...@apache.org>.
On 12/11/10 22:29, Vijay wrote:
> It seems like on Windows java doesn't work well with cygwin-style paths on
> classpath (/cygdrive/d/). The PlatformName error at the beginning is due to
> that. This is coming from the bin/hadoop-config.sh script which is using
> cygwin-styles paths for the jar files.
Windows Java isn't cygwin aware, so it only likes native paths in any
path it sees.
Re: Hadoop installation on Windows
Posted by Steve Loughran <st...@apache.org>.
On 13/11/10 23:34, Christopher Worley wrote:
> Thanks for the advice, guys. I found this tutorial that covers
> installation of 0.21.0:
> http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/
>
> The author suggests adding the "CLASSPATH=`cygpath -wp "$CLASSPATH"`"
> to bin/hadoop-config.sh just like you suggested. I made that change
> and then checked the version. Here's what I got:
>
>
> $ bin/hadoop version
> cygwin warning:
> MS-DOS style path detected: C:\cygwin\usr\local\hadoop-0.21.0\/build/native
> Preferred POSIX equivalent is: /usr/local/hadoop-0.21.0/build/native
> CYGWIN environment variable option "nodosfilewarning" turns off this warning.
> Consult the user's guide for more details about POSIX paths:
> http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
> Hadoop 0.21.0
> Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -
> r 985326
> Compiled by tomwhite on Tue Aug 17 01:02:28 EDT 2010
> From source with checksum a1aeb15b4854808d152989ba76f90fac
>
>
> It gives a warning on the path format, but it appears to work--it
> gives the correct Hadoop version.
>
> When I try to do the next step, formatting the namenode, I get the
> following exceptions:
> http://pastebin.com/YhZ8JZjG
>
> When I try to run "bin/start-dfs.sh" or "bin/start-mapred.sh" I get
> "Hadoop common not found".
>
> I appreciate any help.
If you can get the classpath printed out, it may be that
hadoop-common.jar isn't on it.
As an aside, based on my experience on other projects, the shell scripts
to start java apps are an extreme source of pain, for various reasons
-shell script engine variations on Unix platforms (bash, unix sh)
-bash isn't that good a language for complex development (at least by
java programmers)
-cygwin causes extra confusion
-zero/not enough testing of the scripts in the unit tests
For the limited number of lines, they generate a lot of support calls.
I prefer using python as a launcher, as it's more consistent. Hadoop may
benefit from a python launcher, although Hadoop will probably still
expect cygwin on win32 just for the shell commands it likes to execute.
The other option, assuming the cluster is just for local development, is
have an ant build file to do the classpath setup. They have done the
windows and cygwin support [1], and if you use <java> with
failonerror=true and failonerror=false the ant process will stay hooked
up (so streaming console output both ways), and pass up failures to the
caller.
Just a thought.
-steve
[1] http://ant.apache.org/manual/running.html
Re: Hadoop installation on Windows
Posted by Christopher Worley <cw...@mymail.mines.edu>.
Thanks for the advice, guys. I found this tutorial that covers
installation of 0.21.0:
http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/
The author suggests adding the "CLASSPATH=`cygpath -wp "$CLASSPATH"`"
to bin/hadoop-config.sh just like you suggested. I made that change
and then checked the version. Here's what I got:
$ bin/hadoop version
cygwin warning:
MS-DOS style path detected: C:\cygwin\usr\local\hadoop-0.21.0\/build/native
Preferred POSIX equivalent is: /usr/local/hadoop-0.21.0/build/native
CYGWIN environment variable option "nodosfilewarning" turns off this warning.
Consult the user's guide for more details about POSIX paths:
http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
Hadoop 0.21.0
Subversion https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -
r 985326
Compiled by tomwhite on Tue Aug 17 01:02:28 EDT 2010
>From source with checksum a1aeb15b4854808d152989ba76f90fac
It gives a warning on the path format, but it appears to work--it
gives the correct Hadoop version.
When I try to do the next step, formatting the namenode, I get the
following exceptions:
http://pastebin.com/YhZ8JZjG
When I try to run "bin/start-dfs.sh" or "bin/start-mapred.sh" I get
"Hadoop common not found".
I appreciate any help.
-Chris
On Fri, Nov 12, 2010 at 3:29 PM, Vijay <te...@gmail.com> wrote:
> It seems like on Windows java doesn't work well with cygwin-style paths on
> classpath (/cygdrive/d/). The PlatformName error at the beginning is due to
> that. This is coming from the bin/hadoop-config.sh script which is using
> cygwin-styles paths for the jar files.
>
> The bin/hadoop script works correctly since it has this extra command for
> cywin:
> CLASSPATH=`cygpath -p -w "$CLASSPATH"`
>
> If the above line is added to the bin/hadoop-config.sh, the platform error
> goes away.
>
> In the end I'm getting an error like "Bad connection to FS. command
> aborted." when I try to use the fs command. I'm still figuring out why that
> is.
>
> On Fri, Nov 12, 2010 at 12:09 PM, Jamie Cockrill
> <ja...@gmail.com>wrote:
>
>> Can't spot anything obvious that would prevent it working... can you
>> paste in the other exceptions? Or put them in pastebin
>> (http://pastebin.com/) and link to them here.
>>
>> Also, if you're just trying it out to see what it can do, perhaps
>> using something like Cloudera's training VM might be worthwhile:
>>
>> http://www.cloudera.com/downloads/virtual-machine/
>>
>> Start it up with VMWare player and you have yourself a little test-bed
>> to work with.
>>
>> Ta
>>
>> Jamie
>>
>> On 12 November 2010 20:03, Jamie Cockrill <ja...@gmail.com>
>> wrote:
>> > Very quick spot:
>> >
>> > From the docs:
>> http://hadoop.apache.org/common/docs/current/commands_manual.html#version
>> > ------
>> > version
>> >
>> > Prints the version.
>> >
>> > Usage: hadoop version
>> > ------
>> >
>> > you don't need the '--' on the front of the version argument. Will
>> > have a look at the other thing.
>> >
>> > Ta
>> >
>> > Jamie
>> >
>> >
>> > On 11 November 2010 17:36, Christopher Worley
>> > <ch...@gmail.com> wrote:
>> >> Hello,
>> >>
>> >> I'm trying to get Hadoop working on Cygwin/Windows XP in
>> >> Pseudo-Distributed Mode. I downloaded version 0.21.0 and unpacked it
>> >> to a cygwin directory. I'm following the quickstart directions here:
>> >> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
>> >>
>> >>
>> >> I get the following error when I try to check the Hadoop version:
>> >>
>> >>
>> >> $ bin/hadoop --version
>> >> Exception in thread "main" java.lang.NoClassDefFoundError:
>> >> org/apache/hadoop/util/PlatformName
>> >> Caused by: java.lang.ClassNotFoundException:
>> org.apache.hadoop.util.PlatformName
>> >>
>> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> >> Could not find the main class: org.apache.hadoop.util.PlatformName.
>> Program wil
>> >> l exit.
>> >> Unrecognized option: --version
>> >> Could not create the Java virtual machine.
>> >>
>> >>
>> >> I get similar exceptions when I run "start-all.sh", but I am still
>> >> able to browse the web interface for NameNode and JobTracker.
>> >>
>> >> I'm sure has something to do with an installation step that I'm
>> >> missing--some step that windows users have to do, that linux users do
>> >> not. I did web searches for this problem, but I only found it
>> >> occurring when people were building Hadoop. I'm not building.
>> >>
>> >> I appreciate any help anyone can give.
>> >>
>> >> Thanks,
>> >> Chris
>> >>
>> >
>>
>
Re: Hadoop installation on Windows
Posted by Vijay <te...@gmail.com>.
It seems like on Windows java doesn't work well with cygwin-style paths on
classpath (/cygdrive/d/). The PlatformName error at the beginning is due to
that. This is coming from the bin/hadoop-config.sh script which is using
cygwin-styles paths for the jar files.
The bin/hadoop script works correctly since it has this extra command for
cywin:
CLASSPATH=`cygpath -p -w "$CLASSPATH"`
If the above line is added to the bin/hadoop-config.sh, the platform error
goes away.
In the end I'm getting an error like "Bad connection to FS. command
aborted." when I try to use the fs command. I'm still figuring out why that
is.
On Fri, Nov 12, 2010 at 12:09 PM, Jamie Cockrill
<ja...@gmail.com>wrote:
> Can't spot anything obvious that would prevent it working... can you
> paste in the other exceptions? Or put them in pastebin
> (http://pastebin.com/) and link to them here.
>
> Also, if you're just trying it out to see what it can do, perhaps
> using something like Cloudera's training VM might be worthwhile:
>
> http://www.cloudera.com/downloads/virtual-machine/
>
> Start it up with VMWare player and you have yourself a little test-bed
> to work with.
>
> Ta
>
> Jamie
>
> On 12 November 2010 20:03, Jamie Cockrill <ja...@gmail.com>
> wrote:
> > Very quick spot:
> >
> > From the docs:
> http://hadoop.apache.org/common/docs/current/commands_manual.html#version
> > ------
> > version
> >
> > Prints the version.
> >
> > Usage: hadoop version
> > ------
> >
> > you don't need the '--' on the front of the version argument. Will
> > have a look at the other thing.
> >
> > Ta
> >
> > Jamie
> >
> >
> > On 11 November 2010 17:36, Christopher Worley
> > <ch...@gmail.com> wrote:
> >> Hello,
> >>
> >> I'm trying to get Hadoop working on Cygwin/Windows XP in
> >> Pseudo-Distributed Mode. I downloaded version 0.21.0 and unpacked it
> >> to a cygwin directory. I'm following the quickstart directions here:
> >> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
> >>
> >>
> >> I get the following error when I try to check the Hadoop version:
> >>
> >>
> >> $ bin/hadoop --version
> >> Exception in thread "main" java.lang.NoClassDefFoundError:
> >> org/apache/hadoop/util/PlatformName
> >> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.util.PlatformName
> >>
> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> >> Could not find the main class: org.apache.hadoop.util.PlatformName.
> Program wil
> >> l exit.
> >> Unrecognized option: --version
> >> Could not create the Java virtual machine.
> >>
> >>
> >> I get similar exceptions when I run "start-all.sh", but I am still
> >> able to browse the web interface for NameNode and JobTracker.
> >>
> >> I'm sure has something to do with an installation step that I'm
> >> missing--some step that windows users have to do, that linux users do
> >> not. I did web searches for this problem, but I only found it
> >> occurring when people were building Hadoop. I'm not building.
> >>
> >> I appreciate any help anyone can give.
> >>
> >> Thanks,
> >> Chris
> >>
> >
>
Re: Hadoop installation on Windows
Posted by Jamie Cockrill <ja...@gmail.com>.
Can't spot anything obvious that would prevent it working... can you
paste in the other exceptions? Or put them in pastebin
(http://pastebin.com/) and link to them here.
Also, if you're just trying it out to see what it can do, perhaps
using something like Cloudera's training VM might be worthwhile:
http://www.cloudera.com/downloads/virtual-machine/
Start it up with VMWare player and you have yourself a little test-bed
to work with.
Ta
Jamie
On 12 November 2010 20:03, Jamie Cockrill <ja...@gmail.com> wrote:
> Very quick spot:
>
> From the docs: http://hadoop.apache.org/common/docs/current/commands_manual.html#version
> ------
> version
>
> Prints the version.
>
> Usage: hadoop version
> ------
>
> you don't need the '--' on the front of the version argument. Will
> have a look at the other thing.
>
> Ta
>
> Jamie
>
>
> On 11 November 2010 17:36, Christopher Worley
> <ch...@gmail.com> wrote:
>> Hello,
>>
>> I'm trying to get Hadoop working on Cygwin/Windows XP in
>> Pseudo-Distributed Mode. I downloaded version 0.21.0 and unpacked it
>> to a cygwin directory. I'm following the quickstart directions here:
>> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
>>
>>
>> I get the following error when I try to check the Hadoop version:
>>
>>
>> $ bin/hadoop --version
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> org/apache/hadoop/util/PlatformName
>> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>> Could not find the main class: org.apache.hadoop.util.PlatformName. Program wil
>> l exit.
>> Unrecognized option: --version
>> Could not create the Java virtual machine.
>>
>>
>> I get similar exceptions when I run "start-all.sh", but I am still
>> able to browse the web interface for NameNode and JobTracker.
>>
>> I'm sure has something to do with an installation step that I'm
>> missing--some step that windows users have to do, that linux users do
>> not. I did web searches for this problem, but I only found it
>> occurring when people were building Hadoop. I'm not building.
>>
>> I appreciate any help anyone can give.
>>
>> Thanks,
>> Chris
>>
>
Re: Hadoop installation on Windows
Posted by Jamie Cockrill <ja...@gmail.com>.
Very quick spot:
>From the docs: http://hadoop.apache.org/common/docs/current/commands_manual.html#version
------
version
Prints the version.
Usage: hadoop version
------
you don't need the '--' on the front of the version argument. Will
have a look at the other thing.
Ta
Jamie
On 11 November 2010 17:36, Christopher Worley
<ch...@gmail.com> wrote:
> Hello,
>
> I'm trying to get Hadoop working on Cygwin/Windows XP in
> Pseudo-Distributed Mode. I downloaded version 0.21.0 and unpacked it
> to a cygwin directory. I'm following the quickstart directions here:
> http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html
>
>
> I get the following error when I try to check the Hadoop version:
>
>
> $ bin/hadoop --version
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/util/PlatformName
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
> Could not find the main class: org.apache.hadoop.util.PlatformName. Program wil
> l exit.
> Unrecognized option: --version
> Could not create the Java virtual machine.
>
>
> I get similar exceptions when I run "start-all.sh", but I am still
> able to browse the web interface for NameNode and JobTracker.
>
> I'm sure has something to do with an installation step that I'm
> missing--some step that windows users have to do, that linux users do
> not. I did web searches for this problem, but I only found it
> occurring when people were building Hadoop. I'm not building.
>
> I appreciate any help anyone can give.
>
> Thanks,
> Chris
>