You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Harel Gliksman <ha...@gmail.com> on 2014/10/20 16:59:07 UTC

UDF related: org.apache.hive.com.esotericsoftware.kryo.KryoException

Hello,

I am experiencing inconsistent behavior when trying to use UDF on 0.13.1 on
Amazon's EMR (AMI 3.2.1).

I generated a uber jar and deployed a UDF like so:

create temporary function someFunction as "hive.udf.localization.MyUDF"
using jar "s3://waze.mapreduce.shared/scripts/Hive/MyHive.jar";

I am having these 2 (related?) problems:

1) When I simply try to use my UDF I get

Error: java.lang.RuntimeException:
org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException
Serialization trace:
childRectangles (SomeClass1)
statesTree (SomeClass2)
states (SomeClass3)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:360)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:271)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:431)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException...

After digging I found this:

http://mail-archives.apache.org/mod_mbox/hive-dev/201408.mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E

*and after setting *

*hive.plan.serialization.format=javaXML*

*the UDF is running OK on my test data set of 50 lines.*

2) When running the UDF in a more complex 2-joins query I am getting a
somewhat related error:

org.apache.hive.com.esotericsoftware.kryo.KryoException:
java.lang.NullPointerException
Serialization trace:
org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
Error: java.lang.NullPointerException
Serialization trace:
childRectangles (SomeClass1)
statesTree (SomeClass2)
states (SomeClass3)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator)
opParseCtxMap (org.apache.hadoop.hive.ql.plan.MapWork)
mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)...

*This does not go away even after
setting hive.plan.serialization.format=javaXML *

Can someone please advise?
Many thanks,
Harel.

Re: UDF related: org.apache.hive.com.esotericsoftware.kryo.KryoException

Posted by Navis류승우 <na...@nexr.com>.
The states in UDF(which is represented as "states SomeClass3") are needed?
If it's not, you can try them mark as a transient field.

Thanks,
Navis

2014-10-20 23:59 GMT+09:00 Harel Gliksman <ha...@gmail.com>:

> Hello,
>
> I am experiencing inconsistent behavior when trying to use UDF on 0.13.1
> on Amazon's EMR (AMI 3.2.1).
>
> I generated a uber jar and deployed a UDF like so:
>
> create temporary function someFunction as "hive.udf.localization.MyUDF"
> using jar "s3://waze.mapreduce.shared/scripts/Hive/MyHive.jar";
>
> I am having these 2 (related?) problems:
>
> 1) When I simply try to use my UDF I get
>
> Error: java.lang.RuntimeException:
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException
> Serialization trace:
> childRectangles (SomeClass1)
> statesTree (SomeClass2)
> states (SomeClass3)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:360)
> at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:271)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:431)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:410)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException...
>
> After digging I found this:
>
>
> http://mail-archives.apache.org/mod_mbox/hive-dev/201408.mbox/%3CJIRA.12733732.1407927053435.81293.1408008733570@arcas%3E
>
> *and after setting *
>
> *hive.plan.serialization.format=javaXML*
>
> *the UDF is running OK on my test data set of 50 lines.*
>
> 2) When running the UDF in a more complex 2-joins query I am getting a
> somewhat related error:
>
> org.apache.hive.com.esotericsoftware.kryo.KryoException:
> java.lang.NullPointerException
> Serialization trace:
> org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task
> Error: java.lang.NullPointerException
> Serialization trace:
> childRectangles (SomeClass1)
> statesTree (SomeClass2)
> states (SomeClass3)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator)
> opParseCtxMap (org.apache.hadoop.hive.ql.plan.MapWork)
> mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)...
>
> *This does not go away even after
> setting hive.plan.serialization.format=javaXML *
>
> Can someone please advise?
> Many thanks,
> Harel.
>