Hadoop on Windows

[Nick Lothian <nl...@educationau.edu.au>]

Hi,

I'm trying to get Hadoop running on Windows, and I've found that it
isn't exactly a simple process (yes, I'm using Cygwin).

I've had a quick look though the mailing list, and I see the occasional
question about it, generally with the response "use Cygwin", and no
further details.

Is anyone actually successfully running Hadoop on Windows/Cygwin? If so,
have you had similar problems to those outlined below? I think there's a
fair chance the first two of these are environmental local to my
machine, but it would be nice to be sure that if I keep working on this
it will work in the end...

1) Running bin/start-all.sh gives:
: no address associated with name

That turns out to be a Unix vs DOS line endings thing (!). 

Running the following commands fixed that:

dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves

2) Any script using hadoop-daemon.sh failed. This turned out to be
another Unix vs DOS line endings thing. The critical error looked like
this:

<machine-name>: head: cannot open
`/cygdrive/c/dev/prog/hadoop-0.14.1/bin/../logs/hadoop-nlothian-namenode
-<machine-name>\r.out' for reading: No such file or directory

(note that "\r.out" in the logfile name...)

The fix for that was to alter the hadoop-daemon.sh script. Replace:

	export
HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.log 
with 
	export
HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' | sed
s/\r//`.log

and 

	
log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`hostname`.out 
with 
	
log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-`'hostname' |
sed s/\r//`.out

3) Now the start-all.sh script seems to run ok, but the Java processes
which are started on the slave machine (in this case the same machine as
the master) die soon after launching.

If I manually run:
bin/hadoop jobtracker
bin/hadoop tasktracker
bin/hadoop namenode
bin/hadoop datanode

I can succesfully submit jobs. However, when using the start-all.sh
script the submitted jobs fail to connect (because the jobtracker dies).
Has anyone else seen this? Is there a workaround?

Regards
  Nick Lothian

IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
If you think you may not be the intended recipient, or if you have received this e-mail in error, please contact the sender immediately and delete all copies of this e-mail. 
If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose its contents to any other party. 
This email represents the views of the individual sender, which do not necessarily reflect those of education.au limited except where the sender expressly states otherwise. 
It is your responsibility to scan this email and any files transmitted with it for viruses or any other defects. education.au limited will not be liable for any loss, damage or consequence caused directly or indirectly by this email.

RE: Hadoop on Windows

[Nick Lothian <nl...@educationau.edu.au>]

> 
> Nick Lothian wrote:
> > Yes, mine are set for the Windows style (they weren't, but at some 
> > stage I upgraded my Cywin install and they switched).
> > 
> > However if I switch back to Unix style endings all my bash scripts 
> > fail, again with complaints about line endings (I can't 
> seem to win!):
> 
> This is in a fresh cygwin install with a fresh hadoop install?
> 
> Doug
> 

Yes that's right.

I suspect it's more of a Cygwin bug with switching line endings than a
Hadoop bug, though

IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
If you think you may not be the intended recipient, or if you have received this e-mail in error, please contact the sender immediately and delete all copies of this e-mail. 
If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose its contents to any other party. 
This email represents the views of the individual sender, which do not necessarily reflect those of education.au limited except where the sender expressly states otherwise. 
It is your responsibility to scan this email and any files transmitted with it for viruses or any other defects. education.au limited will not be liable for any loss, damage or consequence caused directly or indirectly by this email.

Re: Hadoop on Windows

[Doug Cutting <cu...@apache.org>]

Nick Lothian wrote:
> Yes, mine are set for the Windows style (they weren't, but at some stage
> I upgraded my Cywin install and they switched).
> 
> However if I switch back to Unix style endings all my bash scripts fail,
> again with complaints about line endings (I can't seem to win!):

This is in a fresh cygwin install with a fresh hadoop install?

Doug

RE: Hadoop on Windows

[Nick Lothian <nl...@educationau.edu.au>]

Yes, mine are set for the Windows style (they weren't, but at some stage
I upgraded my Cywin install and they switched).

However if I switch back to Unix style endings all my bash scripts fail,
again with complaints about line endings (I can't seem to win!):

/cygdrive/c/dev/prog/hadoop-0.14.1/bin/hadoop-config.sh: line 4: $'\r':
command not found
/cygdrive/c/dev/prog/hadoop-0.14.1/bin/hadoop-config.sh: line 6: $'\r':
command not found
/cygdrive/c/dev/prog/hadoop-0.14.1/bin/hadoop-config.sh: line 54: syntax
error: unexpected end of file
etc etc

Nick


> -----Original Message-----
> From: Doug Cutting [mailto:cutting@apache.org] 
> Sent: Thursday, October 11, 2007 3:37 AM
> To: hadoop-user@lucene.apache.org
> Subject: Re: Hadoop on Windows
> 
> Nick Lothian wrote:
> > That turns out to be a Unix vs DOS line endings thing (!). 
> > 
> > Running the following commands fixed that:
> > 
> > dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
> > dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves
> 
> That should not be required.  When you install Cygwin, it has 
> a setting for line endings, which I think defaults and should 
> be left as unix-style, not windows style.  Could that be the issue?
> 
> Doug
> 

IMPORTANT: This e-mail, including any attachments, may contain private or confidential information.
If you think you may not be the intended recipient, or if you have received this e-mail in error, please contact the sender immediately and delete all copies of this e-mail. 
If you are not the intended recipient, you must not reproduce any part of this e-mail or disclose its contents to any other party. 
This email represents the views of the individual sender, which do not necessarily reflect those of education.au limited except where the sender expressly states otherwise. 
It is your responsibility to scan this email and any files transmitted with it for viruses or any other defects. education.au limited will not be liable for any loss, damage or consequence caused directly or indirectly by this email.

Re: Hadoop on Windows

[Doug Cutting <cu...@apache.org>]

Nick Lothian wrote:
> That turns out to be a Unix vs DOS line endings thing (!). 
> 
> Running the following commands fixed that:
> 
> dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/masters
> dos2unix.exe /cygdrive/c/dev/prog/hadoop-0.14.1/conf/slaves

That should not be required.  When you install Cygwin, it has a setting 
for line endings, which I think defaults and should be left as 
unix-style, not windows style.  Could that be the issue?

Doug