[jira] [Work stopped] (HIVE-7066) hive-exec jar is missing avro-mapred

["David Chen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HIVE-7066 stopped by David Chen.

> hive-exec jar is missing avro-mapred
> ------------------------------------
>
>                 Key: HIVE-7066
>                 URL: https://issues.apache.org/jira/browse/HIVE-7066
>             Project: Hive
>          Issue Type: Bug
>            Reporter: David Chen
>            Assignee: David Chen
>
> Running a simple query that reads an Avro table caused the following exception to be thrown on the cluster side:
> {code}
> java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:365)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:276)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:445)
> 	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:438)
> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
> 	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:191)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:394)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> Serialization trace:
> outputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)
> aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:942)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:850)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:864)
> 	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:334)
> 	... 13 more
> Caused by: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat
> 	at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
> 	at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:26)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:343)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:336)
> 	at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:56)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:476)
> 	at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:148)
> 	at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:238)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:226)
> 	at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:745)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> 	... 25 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> 	at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> 	at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:32)
> 	... 37 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/avro/io/DatumWriter
> 	at java.lang.Class.getDeclaredFields0(Native Method)
> 	at java.lang.Class.privateGetDeclaredFields(Class.java:2348)
> 	at java.lang.Class.getDeclaredFields(Class.java:1779)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.rebuildCachedFields(FieldSerializer.java:150)
> 	at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.<init>(FieldSerializer.java:109)
> 	... 42 more
> Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumWriter
> 	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
> 	... 47 more
> {code}
> I took a look at the hive-exec jar and found that the Avro core jar was not included, though avro-mapred is included.
> I confirmed that Avro core was included in the Hive 0.12 hive-exec jar. Was there a reason why this was removed in trunk? It seems that this would break the AvroSerDe.



--
This message was sent by Atlassian JIRA
(v6.2#6252)