[HOD] Example script

[Luca <ra...@yahoo.it>]

Hello everyone,
	in http://hadoop.apache.org/core/docs/r0.16.0/hod.html there is no
mention of scripting abilities of HOD, but with version 0.3 I was used
to do something like:

hod -m N -a "run jar foo.jar jar-parameters"

now I see that in 0.4.0 -a refers to logs-collection and option -z
refers to script.

I wrote a two-lines bash script with this code in it:

#!/bin/bash
hadoop --config /home/luca/hod-test jar
/mnt/scratch/hadoop/hadoop-0.16.0-examples.jar wordcount
file:///mnt/scratch/hadoop/test/part-00000 test.hodscript.out

which is launched right after the allocation, but gets stuck on some

08/03/05 12:43:14 INFO ipc.Client: Retrying connect to server:
svr3.com/10.78.36.204:56895. Already tried 10 time(s).
java.net.ConnectException: Connection refused

The script was launched with command:

hod -m 3 -z test-script.sh

Noticeably, if I from the shell I run manually:
$ hod -o "allocate . 4"
$ hadoop --config . jar /mnt/scratch/hadoop/hadoop-0.16.0-examples.jar
wordcount file:///mnt/scratch/hadoop/test/part-00000 test.hodscript.out
$ hod -o "deallocate ."

everything works well, so I'm pretty sure I'm doing a mistake with the
script, but I didn't find any example to work on. I'm not sure what's
changing in regard to ports among the two invocations.

Cheers,
Luca

Re: [HOD] Example script

[Luca <ra...@yahoo.it>]

Hemanth Yamijala wrote:
> Luca,
>>
>> #!/bin/bash
>> hadoop --config /home/luca/hod-test jar
>> /mnt/scratch/hadoop/hadoop-0.16.0-examples.jar wordcount
>> file:///mnt/scratch/hadoop/test/part-00000 test.hodscript.out
>>
> Can you try removing the --config from this script ? While running 
> scripts, HOD automatically allocates a directory and sets 
> HADOOP_CONF_DIR to that. So, you don't need to mention that.
> 

Yep that worked. I was pretty sure I made an attempt without that option 
but maybe there was something else different.

> That said, I have a very important note: we are changing HOD's user 
> interface a bit based on some feedback from some of our users. Before 
> you write too many scripts, it might be worthwhile to wait until Hadoop 
> 0.16.1 is out which will have a more user friendly HOD interface. Please 
> look at H-2861 for all details.
> 

As for now, I'm only going with very basic script so it's not an issue. 
But good to know that juicy stuff is around the corner :)

Thanks a lot,
Luca

> Thanks
> hemanth
> 
> 

Re: [HOD] Example script

[Hemanth Yamijala <yh...@yahoo-inc.com>]

Luca,
>
> #!/bin/bash
> hadoop --config /home/luca/hod-test jar
> /mnt/scratch/hadoop/hadoop-0.16.0-examples.jar wordcount
> file:///mnt/scratch/hadoop/test/part-00000 test.hodscript.out
>
Can you try removing the --config from this script ? While running 
scripts, HOD automatically allocates a directory and sets 
HADOOP_CONF_DIR to that. So, you don't need to mention that.

That said, I have a very important note: we are changing HOD's user 
interface a bit based on some feedback from some of our users. Before 
you write too many scripts, it might be worthwhile to wait until Hadoop 
0.16.1 is out which will have a more user friendly HOD interface. Please 
look at H-2861 for all details.

Thanks
hemanth