Problems using UDFs

[phil young <ph...@gmail.com>]

I'm getting errors while running contributed UDFs in Hive.

The environment is Hadoop 0.20.2 with Hive from the trunk on 9/30/10

I've tried a few UDFs, and get the same results:
1) The reduce oscillates between 0 and 30% before failing
2) I get the errors below


The "create temporary function" succeeds for a properly named class, so it
seems the jar is being found properly.

Here are a few ways I've invoked the UDFs

drop temporary function row_sequence;
create temporary function row_sequence as
'org.apache.hadoop.hive.contrib.udf.UDFRowSequence';

set mapred.reduce.tasks=1;
select row_sequence(), t.* from userid_new t;

set mapred.reduce.tasks=1;
select user_id, row_sequence() as r
from (
  select user_id
  from userid_new_lim10
  order by user_id
) o
order by user_id;


drop temporary function udftest;
create temporary function udftest as
'org.apache.hadoop.hive.contrib.udf.example.UDFExampleAdd';

select t.n, udftest(t.n) from userid_new_lim10 t group by t.n;
select t.n, udftest(t.n) from userid_new_lim10 t order by t.n;



Here's an example of the errors I see:

java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
	... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
	at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
	... 10 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
	... 13 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
	at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:157)
	... 18 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:126)
	at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:80)
	at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:821)
	at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:833)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:351)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:427)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:383)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:113)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:351)
	at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:415)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:351)
	at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:96)
	... 18 more
Caused by: java.lang.NullPointerException
	at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
	... 31 more