You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by prakash sejwani <pr...@gmail.com> on 2010/03/03 11:50:16 UTC

how to creating custom UDFs in hive

Hi All,
       I follow this link http://wiki.apache.org/hadoop/Hive/HivePlugins
     First, to create a new class that extends UDF, with one or more methods
named evaluate
 Once hive is started up with your jars in the classpath, the final step is
to register your function:

create temporary function ip_to_country as 'com.example.hive.udf.IpToCountry';

it gives me the error the following error


Exception in thread "main" java.lang.NoClassDefFoundError:
com/maxmind/geoip/LookupService
    at com.econify.geoip.IpToCountry.<clinit>(IpToCountry.java:15)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.hive.ql.exec.FunctionTask.getUdfClass(FunctionTask.java:118)
    at
org.apache.hadoop.hive.ql.exec.FunctionTask.createFunction(FunctionTask.java:74)
    at
org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:62)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
    at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: com.maxmind.geoip.LookupService
    at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
    at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
    ... 19 more


I have attached my java file with it.

Re: how to creating custom UDFs in hive

Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Mar 3, 2010 at 10:07 AM, Edward Capriolo <ed...@gmail.com> wrote:
> On Wed, Mar 3, 2010 at 8:11 AM, Sonal Goyal <so...@gmail.com> wrote:
>> Is your com.maxmind.geoip.LookupService class part of your jar?
>> Thanks and Regards,
>> Sonal
>>
>>
>> On Wed, Mar 3, 2010 at 4:20 PM, prakash sejwani <pr...@gmail.com>
>> wrote:
>>>
>>> Hi All,
>>>        I follow this link http://wiki.apache.org/hadoop/Hive/HivePlugins
>>>      First, to create a new class that extends UDF, with one or more
>>> methods named evaluate
>>>  Once hive is started up with your jars in the classpath, the final step
>>> is to register your function:
>>>
>>> create temporary function ip_to_country as
>>> 'com.example.hive.udf.IpToCountry';
>>>
>>> it gives me the error the following error
>>>
>>>
>>>
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/maxmind/geoip/LookupService
>>>     at com.econify.geoip.IpToCountry.<clinit>(IpToCountry.java:15)
>>>     at java.lang.Class.forName0(Native Method)
>>>     at java.lang.Class.forName(Class.java:247)
>>>     at
>>> org.apache.hadoop.hive.ql.exec.FunctionTask.getUdfClass(FunctionTask.java:118)
>>>     at
>>> org.apache.hadoop.hive.ql.exec.FunctionTask.createFunction(FunctionTask.java:74)
>>>     at
>>> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:62)
>>>     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>>>     at
>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>>>     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>>>     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>>>     at
>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>     at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> Caused by: java.lang.ClassNotFoundException:
>>> com.maxmind.geoip.LookupService
>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>>     at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>>     ... 19 more
>>>
>>>
>>> I have attached my java file with it.
>>>
>>
>>
> I have just released similar code. You may be interested in:
>
> http://www.jointhegrid.com/hive-udf-geo-ip-jtg/index.jsp
>

To solve your direct problem, if your UDF depends on a third party jar
be sure to add that with the 'add jar' command.

My usage is like this:
add file GeoIP.dat;
add jar geo-ip-java.jar;
add jar hive-udf-geo-ip-jtg.jar;
create temporary function geoip as 'com.jointhegrid.hive.udf.GenericUDFGeoIP';
select geoip(first, 'COUNTRY_NAME',  './GeoIP.dat' ) from a;

Where geo-ip-java.jar contains the maxmind Lookup Service.

Re: how to creating custom UDFs in hive

Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Mar 3, 2010 at 8:11 AM, Sonal Goyal <so...@gmail.com> wrote:
> Is your com.maxmind.geoip.LookupService class part of your jar?
> Thanks and Regards,
> Sonal
>
>
> On Wed, Mar 3, 2010 at 4:20 PM, prakash sejwani <pr...@gmail.com>
> wrote:
>>
>> Hi All,
>>        I follow this link http://wiki.apache.org/hadoop/Hive/HivePlugins
>>      First, to create a new class that extends UDF, with one or more
>> methods named evaluate
>>  Once hive is started up with your jars in the classpath, the final step
>> is to register your function:
>>
>> create temporary function ip_to_country as
>> 'com.example.hive.udf.IpToCountry';
>>
>> it gives me the error the following error
>>
>>
>>
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> com/maxmind/geoip/LookupService
>>     at com.econify.geoip.IpToCountry.<clinit>(IpToCountry.java:15)
>>     at java.lang.Class.forName0(Native Method)
>>     at java.lang.Class.forName(Class.java:247)
>>     at
>> org.apache.hadoop.hive.ql.exec.FunctionTask.getUdfClass(FunctionTask.java:118)
>>     at
>> org.apache.hadoop.hive.ql.exec.FunctionTask.createFunction(FunctionTask.java:74)
>>     at
>> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:62)
>>     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>>     at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>>     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>>     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>>     at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.lang.ClassNotFoundException:
>> com.maxmind.geoip.LookupService
>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>>     at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>>     ... 19 more
>>
>>
>> I have attached my java file with it.
>>
>
>
I have just released similar code. You may be interested in:

http://www.jointhegrid.com/hive-udf-geo-ip-jtg/index.jsp

Re: how to creating custom UDFs in hive

Posted by Sonal Goyal <so...@gmail.com>.
Is your com.maxmind.geoip.LookupService class part of your jar?
Thanks and Regards,
Sonal


On Wed, Mar 3, 2010 at 4:20 PM, prakash sejwani <pr...@gmail.com>wrote:

> Hi All,
>        I follow this link http://wiki.apache.org/hadoop/Hive/HivePlugins
>      First, to create a new class that extends UDF, with one or more
> methods named evaluate
>  Once hive is started up with your jars in the classpath, the final step is
> to register your function:
>
> create temporary function ip_to_country as 'com.example.hive.udf.IpToCountry';
>
> it gives me the error the following error
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/maxmind/geoip/LookupService
>     at com.econify.geoip.IpToCountry.<clinit>(IpToCountry.java:15)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:247)
>     at
> org.apache.hadoop.hive.ql.exec.FunctionTask.getUdfClass(FunctionTask.java:118)
>     at
> org.apache.hadoop.hive.ql.exec.FunctionTask.createFunction(FunctionTask.java:74)
>     at
> org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:62)
>     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>     at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException:
> com.maxmind.geoip.LookupService
>     at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
>     at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
>     ... 19 more
>
>
> I have attached my java file with it.
>
>