Hadoop in Web applications

["Kilbride, James P." <Ja...@gd-ais.com>]

All,

What's the preferred way to use hadoop within java code running as a web
service in something like tomcat? Would you edit the tomcat
configuration to have access to the hadoop directory? Or simply include
the hadoop jar and have the web service set some configuration somehow
before calling submit(assuming the webservice implements the Tool
interface)?

James Kilbride

Re: Hadoop in Web applications

[Steve Loughran <st...@apache.org>]

Kilbride, James P. wrote:
> All,
> 
> What's the preferred way to use hadoop within java code running as a web
> service in something like tomcat? Would you edit the tomcat
> configuration to have access to the hadoop directory? Or simply include
> the hadoop jar and have the web service set some configuration somehow
> before calling submit(assuming the webservice implements the Tool
> interface)?

1. run hadoop services outside of the web server process,
2. bring up the Hadoop client libraries passing in a jobconf that binds 
to the filesystem/JT in the cluster
3. talk to the things remotely as you desire

there's no need to do the tool interface, just talk using the DfsClient 
and Filesystem client APIs.


e.g to submit work, call JobClient.submitJob(jobConf) on a client you've 
created, to work with a dfs given its URI and a conf file:

             FileSystem dfs = FileSystem.newInstance(uri, conf);
             dfs.initialize(uri, conf);

Re: Hadoop in Web applications

[Venkatesh S <sv...@yahoo-inc.com>]

Latter should work well.

Venkatesh


On 6/17/10 1:04 AM, "Kilbride, James P." <Ja...@gd-ais.com> wrote:

All,

What's the preferred way to use hadoop within java code running as a web
service in something like tomcat? Would you edit the tomcat
configuration to have access to the hadoop directory? Or simply include
the hadoop jar and have the web service set some configuration somehow
before calling submit(assuming the webservice implements the Tool
interface)?

James Kilbride