You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Edward Capriolo <ed...@gmail.com> on 2009/05/01 04:52:40 UTC
Re: Is this the proper usage for a UDF (getting exception)
On Thu, Apr 30, 2009 at 3:11 PM, Zheng Shao <zs...@gmail.com> wrote:
> Putting the jar in hive/lib is not enough - this only makes sure the
> local hive job client can access it.
>
> We also need to make sure remote hive map/reduce tasks can access the jar.
>
> Can you try starting hive with bin/hive -auxjars /xxx/yyy/myudf.jar
>
> Zheng
>
>
>
> On 4/30/09, Edward Capriolo <ed...@gmail.com> wrote:
>> On Thu, Apr 30, 2009 at 12:41 PM, Namit Jain <nj...@facebook.com> wrote:
>>> It looks OK - what is the error that you are getting ?
>>>
>>> -----Original Message-----
>>> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
>>> Sent: Thursday, April 30, 2009 7:31 AM
>>> To: hive-user@hadoop.apache.org
>>> Subject: Is this the proper usage for a UDF (getting exception)
>>>
>>> (I realized i posted this to the dev list, probably better on the user
>>> list)
>>> I made my own UDF, but I am having trouble getting it running . Any hints?
>>> Thanks
>>>
>>> [hadoop@hadoop1 ~]$ /opt/hive/bin/hive -hiveconf
>>> hive.root.logger=INFO,console
>>>
>>> Hive history
>>> file=/tmp/hadoop/hive_job_log_hadoop_200904292045_263570529.txt
>>> 09/04/29 20:45:28 INFO exec.HiveHistory: Hive history
>>> file=/tmp/hadoop/hive_job_log_hadoop_200904292045_263570529.txt
>>> hive> create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO';
>>> 09/04/29 20:45:34 INFO parse.ParseDriver: Parsing command:
>>> create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO'
>>> 09/04/29 20:45:34 INFO parse.ParseDriver: Parse Completed
>>> 09/04/29 20:45:34 INFO parse.FunctionSemanticAnalyzer: analyze done
>>> 09/04/29 20:45:34 INFO ql.Driver: Semantic Analysis Completed
>>> 09/04/29 20:45:34 INFO ql.Driver: Starting command:
>>> create temporary function UDFGEO as 'com.xxx.sysops.udf.UDFGEO'
>>> geoip
>>> OK
>>> 09/04/29 20:45:34 INFO ql.Driver: OK
>>> Time taken: 0.421 seconds
>>> 09/04/29 20:45:34 INFO CliDriver: Time taken: 0.421 seconds
>>>
>>> 09/04/29 20:46:02 INFO CliDriver: Time taken: 0.06 seconds
>>> hive> select UDFGEO('209.191.139.200') from bigdata limit 1 ;
>>> 09/04/29 20:46:09 INFO parse.ParseDriver: Parsing command:
>>> select UDFGEO('209.191.139.200') from bigdata limit 1
>>> 09/04/29 20:46:09 INFO parse.ParseDriver: Parse Completed
>>> 09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
>>> 09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Completed phase 1 of
>>> Semantic Analysis
>>> 09/04/29 20:46:09 INFO parse.SemanticAnalyzer: Get metadata for source
>>> tables
>>> 09/04/29 20:46:09 INFO metastore.HiveMetaStore: 0: Opening raw store
>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>>> 09/04/29 20:46:09 INFO metastore.ObjectStore: ObjectStore, initialize
>>> called
>>> 09/04/29 20:46:09 INFO metastore.ObjectStore: found resource
>>> jpox.properties at file:/opt/hive/conf/jpox.properties
>>> 09/04/29 20:46:12 INFO metastore.ObjectStore: Initialized ObjectStore
>>> 09/04/29 20:46:12 INFO metastore.HiveMetaStore: 0: get_table :
>>> db=default tbl=bigdata
>>> 09/04/29 20:46:13 INFO hive.log: DDL: struct bigdata { double charge,
>>> double clus, double dst, double enumber, double etime, double hist,
>>> double nlb, double qxb, double rnumber, double tracks, double vertex,
>>> double zdc}
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for
>>> destination tables
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed getting
>>> MetaData in Semantic Analysis
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for source
>>> tables
>>> 09/04/29 20:46:13 INFO metastore.HiveMetaStore: 0: get_table :
>>> db=default tbl=bigdata
>>> 09/04/29 20:46:13 INFO metastore.HiveMetaStore: 0: Opening raw store
>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
>>> 09/04/29 20:46:13 INFO metastore.ObjectStore: ObjectStore, initialize
>>> called
>>> 09/04/29 20:46:13 INFO metastore.ObjectStore: Initialized ObjectStore
>>> 09/04/29 20:46:13 INFO hive.log: DDL: struct bigdata { double charge,
>>> double clus, double dst, double enumber, double etime, double hist,
>>> double nlb, double qxb, double rnumber, double tracks, double vertex,
>>> double zdc}
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Get metadata for
>>> destination tables
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed partition pruning
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed sample pruning
>>> 09/04/29 20:46:13 INFO parse.SemanticAnalyzer: Completed plan generation
>>> 09/04/29 20:46:13 INFO ql.Driver: Semantic Analysis Completed
>>> 09/04/29 20:46:13 INFO ql.Driver: Starting command:
>>> select UDFGEO('209.191.139.200') from bigdata limit 1
>>> Total MapReduce jobs = 1
>>> 09/04/29 20:46:13 INFO ql.Driver: Total MapReduce jobs = 1
>>> Number of reduce tasks is set to 0 since there's no reduce operator
>>> 09/04/29 20:46:13 INFO exec.ExecDriver: Number of reduce tasks is set
>>> to 0 since there's no reduce operator
>>> 09/04/29 20:46:13 INFO exec.ExecDriver: Adding input file
>>> /user/hive/warehouse/bigdata
>>> 09/04/29 20:46:13 WARN mapred.JobClient: Use GenericOptionsParser for
>>> parsing the arguments. Applications should implement Tool for the
>>> same.
>>> 09/04/29 20:46:13 INFO mapred.FileInputFormat: Total input paths to
>>> process : 1
>>> Starting Job = job_200904292034_0004, Tracking URL =
>>> http://hadoop1.jointhegrid.local:50030/jobdetails.jsp?jobid=job_200904292034_0004
>>> 09/04/29 20:46:14 INFO exec.ExecDriver: Starting Job =
>>> job_200904292034_0004, Tracking URL =
>>> http://hadoop1.jointhegrid.local:50030/jobdetails.jsp?jobid=job_200904292034_0004
>>> Kill Command = /opt/hadoop/hadoop-0.19.0/bin/../bin/hadoop job
>>> -Dmapred.job.tracker=hadoop1.jointhegrid.local:54311 -kill
>>> job_200904292034_0004
>>> 09/04/29 20:46:14 INFO exec.ExecDriver: Kill Command =
>>> /opt/hadoop/hadoop-0.19.0/bin/../bin/hadoop job
>>> -Dmapred.job.tracker=hadoop1.jointhegrid.local:54311 -kill
>>> job_200904292034_0004
>>> map = 0%, reduce =0%
>>> 09/04/29 20:46:15 INFO exec.ExecDriver: map = 0%, reduce =0%
>>> map = 100%, reduce =100%
>>> 09/04/29 20:46:43 INFO exec.ExecDriver: map = 100%, reduce =100%
>>> Ended Job = job_200904292034_0004 with errors
>>> 09/04/29 20:46:43 ERROR exec.ExecDriver: Ended Job =
>>> job_200904292034_0004 with errors
>>> FAILED: Execution Error, return code 2 from
>>> org.apache.hadoop.hive.ql.exec.ExecDriver
>>> 09/04/29 20:46:43 ERROR ql.Driver: FAILED: Execution Error, return
>>> code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
>>> Time taken: 34.021 seconds
>>> 09/04/29 20:46:43 INFO CliDriver: Time taken: 34.021 seconds
>>> hive>
>>>
>>>
>>>
>>> package com.xxx.sysops.udf;
>>>
>>> import com.maxmind.geoip.LookupService;
>>> import org.apache.hadoop.hive.ql.exec.UDF;
>>> import org.apache.hadoop.io.Text;
>>>
>>> /**
>>> *
>>> * @author ecapriolo
>>> */
>>> public class UDFGEO extends UDF{
>>> static LookupService ls;
>>> static {
>>> try {
>>> ls= new LookupService("/usr/local/share/GeoIP/GeoIPCity.dat",
>>> LookupService.GEOIP_MEMORY_CACHE );
>>> } catch (java.io.IOException ex) { System.err.println("geoip");}
>>> }
>>> Text result= new Text();
>>> public UDFGEO(){
>>>
>>> }
>>> public Text evaluate(Text t){
>>> if (t == null)
>>> return null;
>>> try{
>>> //result.set(ls.getLocation(t.toString()).region);
>>> result.set("blabla");
>>> } catch (Exception ex){System.err.println(ex);}
>>> return result;
>>> }
>>> }
>>>
>>
>> --Everything seems to proceed fine until this:
>>
>> 09/04/29 20:46:43 INFO exec.ExecDriver: map = 100%, reduce =100%
>> Ended Job = job_200904292034_0004 with errors
>> 09/04/29 20:46:43 ERROR exec.ExecDriver: Ended Job =
>> job_200904292034_0004 with errors
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.ExecDriver
>> 09/04/29 20:46:43 ERROR ql.Driver: FAILED: Execution Error, return
>> code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
>> Time taken: 34.021 seconds
>> 09/04/29 20:46:43 INFO CliDriver: Time taken: 34.021 seconds
>>
>> The jar file is in my hive lib. I put it there to make sure there were
>> no classpath issues.
>>
>
> --
> Sent from Gmail for mobile | mobile.google.com
>
> Yours,
> Zheng
>
I created an auxlibs directory and put the jars there. Start up seems
to detect this.
[hadoop@hadoop1 hadoop]$ /opt/hive/bin/hive -hiveconf
hive.root.logger=INFO,console
-hiveconf hive.aux.jars.path=file:///opt/hive/bin/../auxlib/geoip.jar,file:///opt/hive/bin/../auxlib/hadoop-stuff.jar
Hive seems to pick these jars up on startup, but the UDF is only
working when i set mapred.job.tracker=local