running shell commands inside mapper

[souri datta <so...@gmail.com>]

Hi,
 I am trying to run a shell command from within a mapper. The shell command
is of the form:
*  hadoop jar somjarfile arg1 arg2 ...*


Can i do this type of operation from within a mapper?

Also, can i copy the jar file in distributed cache and use it?


Thanks,
Souri

Re: running shell commands inside mapper

[Robert Evans <ev...@yahoo-inc.com>]

Souri,

Yes and no.  Oozie does something like this, but not through the shell, and you can look at how they are doing it.  I don't know the details, but you will probably need to get some delegation tokens to make it work properly if you have security enabled.  You should be able to copy the jar file over in the distributed cache without any problems.  The real question is why are you doing this?  Oozie is doing it to avoid security problems with how it runs user code.  Is this something to launch more jobs in some sort of a loop?  If so you need to be very careful that you don't end up doing a DDOS on the JobTracker by submitting way too many jobs.

--Bobby Evans

On 12/14/11 5:24 AM, "souri datta" <so...@gmail.com> wrote:

Hi,
 I am trying to run a shell command from within a mapper. The shell command is of the form:
  hadoop jar somjarfile arg1 arg2 ...


Can i do this type of operation from within a mapper?

Also, can i copy the jar file in distributed cache and use it?


Thanks,
Souri