Oozie shell action calling hdfs cli

[Matt Goeke <go...@gmail.com>]

All,

Is it possible for an oozie shell action to call the hdfs cli? I have tried
this both with "hadoop distcp <blah> <blah>" as well as "hadoop fs -ls
<blah>" and each time I try it I get the following error:

Error: Exception thrown by the agent :
java.rmi.server.ExportException: Port already in use: <port_value>;
nested exception is:
	java.net.BindException: Address already in use
Failing Oozie Launcher, Main class
[org.apache.oozie.action.hadoop.ShellMain], exit code [1]


I am assuming this is because we are binding <port_value> as the
tasktracker jmx port in the hadoop-env.sh and it is being executed a second
time when it tries to use the CLI. If I am on the right track is there a
simple way around this?

Thanks.

--
Matt

Re: Oozie shell action calling hdfs cli

[Alejandro Abdelnur <tu...@cloudera.com>]

Matt,

this may be tricky:

* in an unsecure setup you'll be shelling out using the UID of the
tasktracker and losing your Hadoop UID
* in a secure setup you'll have to kinit, meaning you need keytabs in
the task node

I'd suggest using Java action, doing FS things natively there and
shelling out afterwards.

Thx

On Fri, Oct 5, 2012 at 10:53 AM, Matt Goeke <go...@gmail.com> wrote:
> 4 days with no response so bumping this just to see if anyone might know
> how to resolve this. I am assuming it is possible to call the hadoop CLI
> from the Oozie shell action right?
>
> On Mon, Oct 1, 2012 at 7:34 PM, Matt Goeke <go...@gmail.com> wrote:
>
>> All,
>>
>> Is it possible for an oozie shell action to call the hdfs cli? I have
>> tried this both with "hadoop distcp <blah> <blah>" as well as "hadoop fs
>> -ls <blah>" and each time I try it I get the following error:
>>
>> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: <port_value>; nested exception is:
>>       java.net.BindException: Address already in use
>> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
>>
>>
>> I am assuming this is because we are binding <port_value> as the
>> tasktracker jmx port in the hadoop-env.sh and it is being executed a second
>> time when it tries to use the CLI. If I am on the right track is there a
>> simple way around this?
>>
>> Thanks.
>>
>> --
>> Matt
>>



-- 
Alejandro

Re: Oozie shell action calling hdfs cli

[Matt Goeke <go...@gmail.com>]

4 days with no response so bumping this just to see if anyone might know
how to resolve this. I am assuming it is possible to call the hadoop CLI
from the Oozie shell action right?

On Mon, Oct 1, 2012 at 7:34 PM, Matt Goeke <go...@gmail.com> wrote:

> All,
>
> Is it possible for an oozie shell action to call the hdfs cli? I have
> tried this both with "hadoop distcp <blah> <blah>" as well as "hadoop fs
> -ls <blah>" and each time I try it I get the following error:
>
> Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: <port_value>; nested exception is:
> 	java.net.BindException: Address already in use
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
>
>
> I am assuming this is because we are binding <port_value> as the
> tasktracker jmx port in the hadoop-env.sh and it is being executed a second
> time when it tries to use the CLI. If I am on the right track is there a
> simple way around this?
>
> Thanks.
>
> --
> Matt
>