Using UDFs stored on HDFS

[Philip Zeyliger <ph...@cloudera.com>]

Hi folks,

I have a quick question about UDF support in Hive.  I'm on the 0.5 branch.
 Can you use a UDF where the jar which contains the function is on HDFS, and
not on the local filesystem.  Specifically, the following does not seem to
work:

# This is Hive 0.5, from svn
$bin/hive
Hive history file=/tmp/philip/hive_job_log_philip_201002081541_370227273.txt
hive> add jar hdfs://localhost/FooTest.jar;

Added hdfs://localhost/FooTest.jar to class path
hive> create temporary function cube as 'com.cloudera.FooTestUDF';

FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.FunctionTask

Does this work for other people?  I could probably fix it by changing "add
jar" to download remote jars locally, when necessary (to load them into the
classpath), or update URLClassLoader (or whatever is underneath there) to
read directly from HDFS, which seems a bit more fragile.  But I wanted to
make sure that my interpretation of what's going on is right before I have
at it.

Thanks,

-- Philip

Re: Using UDFs stored on HDFS

[Zheng Shao <zs...@gmail.com>]

Yes that's correct. I prefer to download the jars in "add jar".

Zheng

On Mon, Feb 8, 2010 at 3:46 PM, Philip Zeyliger <ph...@cloudera.com> wrote:
> Hi folks,
>
> I have a quick question about UDF support in Hive.  I'm on the 0.5 branch.
>  Can you use a UDF where the jar which contains the function is on HDFS, and
> not on the local filesystem.  Specifically, the following does not seem to
> work:
>
> # This is Hive 0.5, from svn
> $bin/hive
> Hive history file=/tmp/philip/hive_job_log_philip_201002081541_370227273.txt
> hive> add jar hdfs://localhost/FooTest.jar;
>
> Added hdfs://localhost/FooTest.jar to class path
> hive> create temporary function cube as 'com.cloudera.FooTestUDF';
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.FunctionTask
>
> Does this work for other people?  I could probably fix it by changing "add
> jar" to download remote jars locally, when necessary (to load them into the
> classpath), or update URLClassLoader (or whatever is underneath there) to
> read directly from HDFS, which seems a bit more fragile.  But I wanted to
> make sure that my interpretation of what's going on is right before I have
> at it.
> Thanks,
> -- Philip



-- 
Yours,
Zheng