You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hivemall.apache.org by "Makoto Yui (JIRA)" <ji...@apache.org> on 2019/01/23 17:49:00 UTC
[jira] [Updated] (HIVEMALL-236) to_json cause
KryoException/NullPointerException with ArrayList due to Kryo bug
[ https://issues.apache.org/jira/browse/HIVEMALL-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Makoto Yui updated HIVEMALL-236:
--------------------------------
Description:
{code:java}
SELECT
from_json(
'{ "person" : { "name" : "makoto" , "age" : 37 } }',
'struct<name:string,age:int>',
array('person')
){code}
to_json cause the following Kryo exception depending on underlying Kryo version.
{code:java}
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
columnNames (hivemall.tools.json.ToJsonUDF)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:915)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:823)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:837)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:330)
... 17 more
Caused by: java.lang.NullPointerException
at java.util.Arrays$ArrayList.size(Arrays.java:3818)
at java.util.AbstractList.add(AbstractList.java:108)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:105)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
... 44 more{code}
BTW, the following query will run without error:
{code:java}
select
from_json(
'[0.1,1.1,2.2]',
'array<double>'
),{code}
was:
{code:java}
select
from_json( '{ "person" : { "name" : "makoto" , "age" : 37 } }', 'struct<name:string,age:int>', array('person') ){code}
to_json cause the following Kryo exception depending on underlying Kryo version.
{code:java}
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
columnNames (hivemall.tools.json.ToJsonUDF)
genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:915)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:823)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:837)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:330)
... 17 more
Caused by: java.lang.NullPointerException
at java.util.Arrays$ArrayList.size(Arrays.java:3818)
at java.util.AbstractList.add(AbstractList.java:108)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:105)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
... 44 more{code}
BTW, the following query will run without error:
{code:java}
select
from_json(
'[0.1,1.1,2.2]',
'array<double>'
),{code}
> to_json cause KryoException/NullPointerException with ArrayList due to Kryo bug
> -------------------------------------------------------------------------------
>
> Key: HIVEMALL-236
> URL: https://issues.apache.org/jira/browse/HIVEMALL-236
> Project: Hivemall
> Issue Type: Bug
> Reporter: Makoto Yui
> Assignee: Makoto Yui
> Priority: Major
>
> {code:java}
> SELECT
> from_json(
> '{ "person" : { "name" : "makoto" , "age" : 37 } }',
> 'struct<name:string,age:int>',
> array('person')
> ){code}
> to_json cause the following Kryo exception depending on underlying Kryo version.
> {code:java}
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
> Serialization trace:
> columnNames (hivemall.tools.json.ToJsonUDF)
> genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)
> colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)
> childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:915)
> at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:823)
> at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:837)
> at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:330)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at java.util.Arrays$ArrayList.size(Arrays.java:3818)
> at java.util.AbstractList.add(AbstractList.java:108)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:105)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
> at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> ... 44 more{code}
> BTW, the following query will run without error:
> {code:java}
> select
> from_json(
> '[0.1,1.1,2.2]',
> 'array<double>'
> ),{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)