You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Mohit Bhatnagar <mo...@adobe.com> on 2015/05/28 09:33:37 UTC

Issue while building CUBE: java.io.IOException: Deserialization error:

Hi,
Can someone help me out with this. I am getting the below error while building the cube in stage 2 :
Here are the logs

2015-05-27 19:04:58,108 FATAL [IPC Server handler 1 on 44765] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1432706588203_2125_m_000000_0 - exited : java.io.IOException: Deserialization error: org.apache.hadoop.hive.metastore.api.Table; local class incompatible: stream classdesc serialVersionUID = 398473631015277182, local class serialVersionUID = -946662244473213550
        at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
        at org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.InvalidClassException: org.apache.hadoop.hive.metastore.api.Table; local class incompatible: stream classdesc serialVersionUID = 398473631015277182, local class serialVersionUID = -946662244473213550
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
        ... 11 more

2015-05-27 19:04:58,108 INFO [IPC Server handler 1 on 44765] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1432706588203_2125_m_000000_0: Error: java.io.IOException: Deserialization error: org.apache.hadoop.hive.metastore.api.Table; local class incompatible: stream classdesc serialVersionUID = 398473631015277182, local class serialVersionUID = -946662244473213550
        at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
        at org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.InvalidClassException: org.apache.hadoop.hive.metastore.api.Table; local class incompatible: stream classdesc serialVersionUID = 398473631015277182, local class serialVersionUID = -946662244473213550
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
        ... 11 more

2015-05-27 19:04:58,110 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1432706588203_2125_m_000000_0: Error: java.io.IOException: Deserialization error: org.apache.hadoop.hive.metastore.api.Table; local class incompatible: stream classdesc serialVersionUID = 398473631015277182, local class serialVersionUID = -946662244473213550
        at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
        at org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.io.InvalidClassException: org.apache.hadoop.hive.metastore.api.Table; local class incompatible: stream classdesc serialVersionUID = 398473631015277182, local class serialVersionUID = -946662244473213550
        at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
        ... 11 more

2015-05-27 19:04:58,112 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2015-05-27 19:04:58,114 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e38_1432706588203_2125_01_000002 taskAttempt attempt_1432706588203_2125_m_000000_0
2015-05-27 19:04:58,114 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1432706588203_2125_m_000000_0
2015-05-27 19:04:58,115 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : sj1dra054.corp.adobe.com:8041
2015-05-27 19:04:58,138 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2015-05-27 19:04:58,139 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2015-05-27 19:04:58,148 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/tmp/kylin-ffb0e3f9-4720-418a-a8d9-924f47fe1eea/mmmmmm/fact_distinct_columns/_temporary/1/_temporary/attempt_1432706588203_2125_m_000000_0
2015-05-27 19:04:58,150 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sj1dra054.corp.adobe.com to /sb
2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved sj1dra151.corp.adobe.com to /sa
2015-05-27 19:04:58,161 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node sj1dra054.corp.adobe.com
2015-05-27 19:04:58,164 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1432706588203_2125_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2015-05-27 19:04:58,165 INFO [Thread-51] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1432706588203_2125_m_000000_1 to list of failed maps
2015-05-27 19:04:58,365 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:1 RackLocal:0
2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1432706588203_2125: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:321024, vCores:262> knownNMs=10
2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:321024, vCores:262>
2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1
2015-05-27 19:04:59,378 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e38_1432706588203_2125_01_000002
2015-05-27 19:04:59,379 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1432706588203_2125_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143




Thanks,
Mohit

Re: Issue while building CUBE: java.io.IOException: Deserialization error:

Posted by "Shi, Shaofeng" <sh...@ebay.com>.
InterestingÅ . Could you run $KYLIN_HOME/bin/find-hive-dependency.sh and
then copy the output here? If there is some entry like hive/lib/*, please
also copy the list of that folder, that might be helpful for identify the
issue.

On 5/28/15, 5:01 PM, "Mohit Bhatnagar" <mo...@adobe.com> wrote:

>Hi,
>
>We have 2 jars hive-metastore-0.13.1-cdh5.3.3.jar and hive-metastore.jar.
>Both of these have org.apache.hadoop.hive.metastore.api.Table class. But
>both their checksums are same. i.e same version. Can there be any other
>issue? 
>
>
>
>Thanks,
>Mohit 
>
>-----Original Message-----
>From: ShaoFeng Shi [mailto:shaofengshi@gmail.com]
>Sent: Thursday, May 28, 2015 2:03 PM
>To: dev@kylin.incubator.apache.org
>Subject: Re: Issue while building CUBE: java.io.IOException:
>Deserialization error:
>
>This error indicates there might be different version jars in your
>environment; As the class is packaged in hive-metastore-*.jar, please
>search your machine to see whether there are different versions, and then
>remove the redundant one;
>
>2015-05-28 15:33 GMT+08:00 Mohit Bhatnagar <mo...@adobe.com>:
>
>> Hi,
>> Can someone help me out with this. I am getting the below error while
>> building the cube in stage 2 :
>> Here are the logs
>>
>> 2015-05-27 19:04:58,108 FATAL [IPC Server handler 1 on 44765]
>> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
>> attempt_1432706588203_2125_m_000000_0 - exited : java.io.IOException:
>> Deserialization error: org.apache.hadoop.hive.metastore.api.Table;
>> local class incompatible: stream classdesc serialVersionUID =
>> 398473631015277182, local class serialVersionUID = -946662244473213550
>>         at
>> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>>         at
>> 
>>org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:13
>>9)
>>         at
>> 
>>org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserialize
>>r.deserialize(WritableSerialization.java:71)
>>         at
>> 
>>org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserialize
>>r.deserialize(WritableSerialization.java:42)
>>         at
>> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>>         at 
>>org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:415)
>>         at
>> 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
>>.java:1642)
>>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> Caused by: java.io.InvalidClassException:
>> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
>> stream classdesc serialVersionUID = 398473631015277182, local class
>> serialVersionUID = -946662244473213550
>>         at
>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>>         at
>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>>         at
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>>         at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>>         at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at 
>>java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>         at
>> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>>         ... 11 more
>>
>> 2015-05-27 19:04:58,108 INFO [IPC Server handler 1 on 44765]
>> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report
>> from
>> attempt_1432706588203_2125_m_000000_0: Error: java.io.IOException:
>> Deserialization error: org.apache.hadoop.hive.metastore.api.Table;
>> local class incompatible: stream classdesc serialVersionUID =
>> 398473631015277182, local class serialVersionUID = -946662244473213550
>>         at
>> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>>         at
>> 
>>org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:13
>>9)
>>         at
>> 
>>org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserialize
>>r.deserialize(WritableSerialization.java:71)
>>         at
>> 
>>org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserialize
>>r.deserialize(WritableSerialization.java:42)
>>         at
>> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>>         at 
>>org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:415)
>>         at
>> 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
>>.java:1642)
>>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> Caused by: java.io.InvalidClassException:
>> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
>> stream classdesc serialVersionUID = 398473631015277182, local class
>> serialVersionUID = -946662244473213550
>>         at
>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>>         at
>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>>         at
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>>         at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>>         at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at 
>>java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>         at
>> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>>         ... 11 more
>>
>> 2015-05-27 19:04:58,110 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>> Diagnostics report from attempt_1432706588203_2125_m_000000_0: Error:
>> java.io.IOException: Deserialization error:
>> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
>> stream classdesc serialVersionUID = 398473631015277182, local class
>> serialVersionUID = -946662244473213550
>>         at
>> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>>         at
>> 
>>org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:13
>>9)
>>         at
>> 
>>org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserialize
>>r.deserialize(WritableSerialization.java:71)
>>         at
>> 
>>org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserialize
>>r.deserialize(WritableSerialization.java:42)
>>         at
>> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>>         at 
>>org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:415)
>>         at
>> 
>>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation
>>.java:1642)
>>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
>> Caused by: java.io.InvalidClassException:
>> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
>> stream classdesc serialVersionUID = 398473631015277182, local class
>> serialVersionUID = -946662244473213550
>>         at
>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>>         at
>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>>         at
>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>>         at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at
>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>>         at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>>         at
>> 
>>java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>>         at
>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>>         at 
>>java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>>         at
>> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>>         ... 11 more
>>
>> 2015-05-27 19:04:58,112 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from
>> RUNNING to FAIL_CONTAINER_CLEANUP
>> 2015-05-27 19:04:58,114 INFO [ContainerLauncher #1]
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
>> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
>> container_e38_1432706588203_2125_01_000002 taskAttempt
>> attempt_1432706588203_2125_m_000000_0
>> 2015-05-27 19:04:58,114 INFO [ContainerLauncher #1]
>> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
>> KILLING
>> attempt_1432706588203_2125_m_000000_0
>> 2015-05-27 19:04:58,115 INFO [ContainerLauncher #1]
>> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
>> Opening proxy : sj1dra054.corp.adobe.com:8041
>> 2015-05-27 19:04:58,138 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from
>> FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
>> 2015-05-27 19:04:58,139 INFO [CommitterEvent Processor #1]
>> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler:
>> Processing the event EventType: TASK_ABORT
>> 2015-05-27 19:04:58,148 WARN [CommitterEvent Processor #1]
>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
>> delete
>> hdfs://nameservice1/tmp/kylin-ffb0e3f9-4720-418a-a8d9-924f47fe1eea/mmm
>> mmm/fact_distinct_columns/_temporary/1/_temporary/attempt_143270658820
>> 3_2125_m_000000_0
>> 2015-05-27 19:04:58,150 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from
>> FAIL_TASK_CLEANUP to FAILED
>> 2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.yarn.util.RackResolver: Resolved
>> sj1dra054.corp.adobe.com to /sb
>> 2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.yarn.util.RackResolver: Resolved
>> sj1dra151.corp.adobe.com to /sa
>> 2015-05-27 19:04:58,161 INFO [Thread-51]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures
>> on node sj1dra054.corp.adobe.com
>> 2015-05-27 19:04:58,164 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>> attempt_1432706588203_2125_m_000000_1 TaskAttempt Transitioned from
>> NEW to UNASSIGNED
>> 2015-05-27 19:04:58,165 INFO [Thread-51]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added
>> attempt_1432706588203_2125_m_000000_1 to list of failed maps
>> 2015-05-27 19:04:58,365 INFO [RMCommunicator Allocator]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
>> Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0
>> AssignedMaps:1
>> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0
>> HostLocal:1 RackLocal:0
>> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
>> getResources() for application_1432706588203_2125: ask=1 release= 0
>> newContainers=0
>> finishedContainers=0 resourcelimit=<memory:321024, vCores:262>
>> knownNMs=10
>> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator:
>> Recalculating schedule, headroom=<memory:321024, vCores:262>
>> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce
>> slow start threshold not met. completedMapsForReduceSlowstart 1
>> 2015-05-27 19:04:59,378 INFO [RMCommunicator Allocator]
>> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
>> completed container container_e38_1432706588203_2125_01_000002
>> 2015-05-27 19:04:59,379 INFO [AsyncDispatcher event handler]
>> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
>> Diagnostics report from attempt_1432706588203_2125_m_000000_0:
>> Container killed by the ApplicationMaster.
>> Container killed on request. Exit code is 143
>>
>>
>>
>>
>> Thanks,
>> Mohit
>>


RE: Issue while building CUBE: java.io.IOException: Deserialization error:

Posted by Mohit Bhatnagar <mo...@adobe.com>.
Hi,

We have 2 jars hive-metastore-0.13.1-cdh5.3.3.jar and hive-metastore.jar. Both of these have org.apache.hadoop.hive.metastore.api.Table class. But both their checksums are same. i.e same version. Can there be any other issue? 



Thanks,
Mohit 

-----Original Message-----
From: ShaoFeng Shi [mailto:shaofengshi@gmail.com] 
Sent: Thursday, May 28, 2015 2:03 PM
To: dev@kylin.incubator.apache.org
Subject: Re: Issue while building CUBE: java.io.IOException: Deserialization error:

This error indicates there might be different version jars in your environment; As the class is packaged in hive-metastore-*.jar, please search your machine to see whether there are different versions, and then remove the redundant one;

2015-05-28 15:33 GMT+08:00 Mohit Bhatnagar <mo...@adobe.com>:

> Hi,
> Can someone help me out with this. I am getting the below error while 
> building the cube in stage 2 :
> Here are the logs
>
> 2015-05-27 19:04:58,108 FATAL [IPC Server handler 1 on 44765]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1432706588203_2125_m_000000_0 - exited : java.io.IOException:
> Deserialization error: org.apache.hadoop.hive.metastore.api.Table; 
> local class incompatible: stream classdesc serialVersionUID = 
> 398473631015277182, local class serialVersionUID = -946662244473213550
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.InvalidClassException:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class 
> serialVersionUID = -946662244473213550
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>         ... 11 more
>
> 2015-05-27 19:04:58,108 INFO [IPC Server handler 1 on 44765]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report 
> from
> attempt_1432706588203_2125_m_000000_0: Error: java.io.IOException:
> Deserialization error: org.apache.hadoop.hive.metastore.api.Table; 
> local class incompatible: stream classdesc serialVersionUID = 
> 398473631015277182, local class serialVersionUID = -946662244473213550
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.InvalidClassException:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class 
> serialVersionUID = -946662244473213550
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>         ... 11 more
>
> 2015-05-27 19:04:58,110 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: 
> Diagnostics report from attempt_1432706588203_2125_m_000000_0: Error:
> java.io.IOException: Deserialization error:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class 
> serialVersionUID = -946662244473213550
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.InvalidClassException:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class 
> serialVersionUID = -946662244473213550
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>         ... 11 more
>
> 2015-05-27 19:04:58,112 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from 
> RUNNING to FAIL_CONTAINER_CLEANUP
> 2015-05-27 19:04:58,114 INFO [ContainerLauncher #1]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e38_1432706588203_2125_01_000002 taskAttempt
> attempt_1432706588203_2125_m_000000_0
> 2015-05-27 19:04:58,114 INFO [ContainerLauncher #1]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: 
> KILLING
> attempt_1432706588203_2125_m_000000_0
> 2015-05-27 19:04:58,115 INFO [ContainerLauncher #1]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : sj1dra054.corp.adobe.com:8041
> 2015-05-27 19:04:58,138 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from 
> FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
> 2015-05-27 19:04:58,139 INFO [CommitterEvent Processor #1]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: 
> Processing the event EventType: TASK_ABORT
> 2015-05-27 19:04:58,148 WARN [CommitterEvent Processor #1]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not 
> delete
> hdfs://nameservice1/tmp/kylin-ffb0e3f9-4720-418a-a8d9-924f47fe1eea/mmm
> mmm/fact_distinct_columns/_temporary/1/_temporary/attempt_143270658820
> 3_2125_m_000000_0
> 2015-05-27 19:04:58,150 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from 
> FAIL_TASK_CLEANUP to FAILED
> 2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved 
> sj1dra054.corp.adobe.com to /sb
> 2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved 
> sj1dra151.corp.adobe.com to /sa
> 2015-05-27 19:04:58,161 INFO [Thread-51]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures 
> on node sj1dra054.corp.adobe.com
> 2015-05-27 19:04:58,164 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_1 TaskAttempt Transitioned from 
> NEW to UNASSIGNED
> 2015-05-27 19:04:58,165 INFO [Thread-51]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added
> attempt_1432706588203_2125_m_000000_1 to list of failed maps
> 2015-05-27 19:04:58,365 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 
> AssignedMaps:1
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0
> HostLocal:1 RackLocal:0
> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 
> getResources() for application_1432706588203_2125: ask=1 release= 0 
> newContainers=0
> finishedContainers=0 resourcelimit=<memory:321024, vCores:262> 
> knownNMs=10
> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 
> Recalculating schedule, headroom=<memory:321024, vCores:262>
> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce 
> slow start threshold not met. completedMapsForReduceSlowstart 1
> 2015-05-27 19:04:59,378 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received 
> completed container container_e38_1432706588203_2125_01_000002
> 2015-05-27 19:04:59,379 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: 
> Diagnostics report from attempt_1432706588203_2125_m_000000_0: 
> Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143
>
>
>
>
> Thanks,
> Mohit
>

Re: Issue while building CUBE: java.io.IOException: Deserialization error:

Posted by ShaoFeng Shi <sh...@gmail.com>.
This error indicates there might be different version jars in your
environment; As the class is packaged in hive-metastore-*.jar, please
search your machine to see whether there are different versions, and then
remove the redundant one;

2015-05-28 15:33 GMT+08:00 Mohit Bhatnagar <mo...@adobe.com>:

> Hi,
> Can someone help me out with this. I am getting the below error while
> building the cube in stage 2 :
> Here are the logs
>
> 2015-05-27 19:04:58,108 FATAL [IPC Server handler 1 on 44765]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1432706588203_2125_m_000000_0 - exited : java.io.IOException:
> Deserialization error: org.apache.hadoop.hive.metastore.api.Table; local
> class incompatible: stream classdesc serialVersionUID = 398473631015277182,
> local class serialVersionUID = -946662244473213550
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.InvalidClassException:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class
> serialVersionUID = -946662244473213550
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>         ... 11 more
>
> 2015-05-27 19:04:58,108 INFO [IPC Server handler 1 on 44765]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> attempt_1432706588203_2125_m_000000_0: Error: java.io.IOException:
> Deserialization error: org.apache.hadoop.hive.metastore.api.Table; local
> class incompatible: stream classdesc serialVersionUID = 398473631015277182,
> local class serialVersionUID = -946662244473213550
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.InvalidClassException:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class
> serialVersionUID = -946662244473213550
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>         ... 11 more
>
> 2015-05-27 19:04:58,110 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1432706588203_2125_m_000000_0: Error:
> java.io.IOException: Deserialization error:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class
> serialVersionUID = -946662244473213550
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.io.InvalidClassException:
> org.apache.hadoop.hive.metastore.api.Table; local class incompatible:
> stream classdesc serialVersionUID = 398473631015277182, local class
> serialVersionUID = -946662244473213550
>         at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
>         at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
>         at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:115)
>         ... 11 more
>
> 2015-05-27 19:04:58,112 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from RUNNING
> to FAIL_CONTAINER_CLEANUP
> 2015-05-27 19:04:58,114 INFO [ContainerLauncher #1]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_e38_1432706588203_2125_01_000002 taskAttempt
> attempt_1432706588203_2125_m_000000_0
> 2015-05-27 19:04:58,114 INFO [ContainerLauncher #1]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1432706588203_2125_m_000000_0
> 2015-05-27 19:04:58,115 INFO [ContainerLauncher #1]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : sj1dra054.corp.adobe.com:8041
> 2015-05-27 19:04:58,138 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from
> FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
> 2015-05-27 19:04:58,139 INFO [CommitterEvent Processor #1]
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing
> the event EventType: TASK_ABORT
> 2015-05-27 19:04:58,148 WARN [CommitterEvent Processor #1]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
> delete
> hdfs://nameservice1/tmp/kylin-ffb0e3f9-4720-418a-a8d9-924f47fe1eea/mmmmmm/fact_distinct_columns/_temporary/1/_temporary/attempt_1432706588203_2125_m_000000_0
> 2015-05-27 19:04:58,150 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_0 TaskAttempt Transitioned from
> FAIL_TASK_CLEANUP to FAILED
> 2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved
> sj1dra054.corp.adobe.com to /sb
> 2015-05-27 19:04:58,161 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved
> sj1dra151.corp.adobe.com to /sa
> 2015-05-27 19:04:58,161 INFO [Thread-51]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on
> node sj1dra054.corp.adobe.com
> 2015-05-27 19:04:58,164 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1432706588203_2125_m_000000_1 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2015-05-27 19:04:58,165 INFO [Thread-51]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added
> attempt_1432706588203_2125_m_000000_1 to list of failed maps
> 2015-05-27 19:04:58,365 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0
> HostLocal:1 RackLocal:0
> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1432706588203_2125: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:321024, vCores:262> knownNMs=10
> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=<memory:321024, vCores:262>
> 2015-05-27 19:04:58,368 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow
> start threshold not met. completedMapsForReduceSlowstart 1
> 2015-05-27 19:04:59,378 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_e38_1432706588203_2125_01_000002
> 2015-05-27 19:04:59,379 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1432706588203_2125_m_000000_0: Container killed by the
> ApplicationMaster.
> Container killed on request. Exit code is 143
>
>
>
>
> Thanks,
> Mohit
>