Running Hadoop without bin/hadoop

[Michael Basnight <mb...@gmail.com>]

I have a java app that runs in tomcat and now needs to talk to my  
hadoop infrastructure. Typically, all the testing ive done / examples  
show starting something that uses hadoop via the 'bin/hadoop -jar'  
cmd, but as you can imagine this is no good for a existing tomcat app.  
Ive looked thru the .sh files in the bin/ dir, and it would require  
extensive work to mod the script to export the env variables so that  
tomcat can be restarted without a special init script (and with those  
variables in tact). The last thing i want to do is hand crank files  
that may or may not change in new hadoop distros. Is there a known way  
to use the hadoop infrastructure outside of the bin/hadoop -jar command?

mb

Re: Running Hadoop without bin/hadoop

[Steve Loughran <st...@apache.org>]

Michael Basnight wrote:
> I have a java app that runs in tomcat and now needs to talk to my hadoop 
> infrastructure. Typically, all the testing ive done / examples show 
> starting something that uses hadoop via the 'bin/hadoop -jar' cmd, but 
> as you can imagine this is no good for a existing tomcat app. Ive looked 
> thru the .sh files in the bin/ dir, and it would require extensive work 
> to mod the script to export the env variables so that tomcat can be 
> restarted without a special init script (and with those variables in 
> tact). The last thing i want to do is hand crank files that may or may 
> not change in new hadoop distros. Is there a known way to use the hadoop 
> infrastructure outside of the bin/hadoop -jar command?
> 
> mb


what are you trying to do? Submit jobs or start hadoop itself?

Hadoop is tricky to start up in-VM; the hadoop-3628 branch of trunk can 
do this, but your security manager needs to intercept the odd call to 
System.exit(), and there are a lot of singletons for monitoring -better 
to start up hadoop in a new VM.

1. JobSubmission can be done remotely by way of the JobClient api

http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/components/submitter/SubmitterImpl.java?view=markup

2. You can also run any instance of Tool by creating then invoking it 
-with any configuration you choose to run

http://smartfrog.svn.sourceforge.net/viewvc/smartfrog/trunk/core/components/hadoop/src/org/smartfrog/services/hadoop/components/submitter/ToolRunnerComponentImpl.java?view=markup

Both suffer from version-sensitivity, everything has to be on exactly 
the same version of hadoop, and the hadoop cluster has to be exposed to 
tomcat, which can be a problem if tomcat is in a DMZ and you are trying 
to secure Hadoop.

Re: Running Hadoop without bin/hadoop

[jason hadoop <ja...@gmail.com>]

There is a decent section on this in the pro hadoop book, basically part of
the unit test section.

You have to define a hadoop.log.dir java property and ensure that the
hadoop-default.xml file and any local parameters you need are available.
Then you can run standalone with no particular trouble.

On Thu, Jul 2, 2009 at 11:33 AM, Michael Basnight <mb...@gmail.com>wrote:

> I have a java app that runs in tomcat and now needs to talk to my hadoop
> infrastructure. Typically, all the testing ive done / examples show starting
> something that uses hadoop via the 'bin/hadoop -jar' cmd, but as you can
> imagine this is no good for a existing tomcat app. Ive looked thru the .sh
> files in the bin/ dir, and it would require extensive work to mod the script
> to export the env variables so that tomcat can be restarted without a
> special init script (and with those variables in tact). The last thing i
> want to do is hand crank files that may or may not change in new hadoop
> distros. Is there a known way to use the hadoop infrastructure outside of
> the bin/hadoop -jar command?
>
> mb
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals