You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by xianbin wang <wa...@gmail.com> on 2015/12/24 07:03:22 UTC

encouter Deserialization error when load hive table

anyone have a idea for such a error,when load hive table, error as follow:

L4J [2015-12-24 13:54:39,354][INFO][org.apache.hadoop.mapreduce.Job] - Task
Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
Error: java.io.IOException: Deserialization error: invalid stream header:
926A2769
	at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:120)
	at
org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:132)
	at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
	at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
	at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:754)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.StreamCorruptedException: invalid stream header: 926A2769
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
	at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
	... 11 more

L4J [2015-12-24 13:54:41,105][INFO][org.apache.hadoop.mapreduce.Job] -  map
0% reduce 0%

--
View this message in context: http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: Re: encouter Deserialization error when load hive table

Posted by xianbin wang <wa...@gmail.com>.
hi yang!

sorry for reply so late, how could hbase be involved in this stage, it seem
like MR job use HCat load data from hive table, it is not reasonable.

--
View this message in context: http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3441.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: Re: encouter Deserialization error when load hive table

Posted by Li Yang <li...@apache.org>.
Kylin 2.0-rc still requires hbase 0.9x, it does not work with HBase 1.x.
That likely be the problem.

On Tue, Jan 12, 2016 at 5:55 PM, wangxianbin1987@gmail.com <
wangxianbin1987@gmail.com> wrote:

> HDP2.3.2, kylin 2.0-rc
>
>
>
> wangxianbin1987@gmail.com
>
> From: Li Yang
> Date: 2016-01-12 17:40
> To: dev
> Subject: Re: Re: encouter Deserialization error when load hive table
> Then let's confirm your Kylin version and HDP version first. What are they?
>
> On Thu, Jan 7, 2016 at 4:03 PM, xianbin wang <wa...@gmail.com>
> wrote:
>
> > hi all!
> >
> > I use HDP sandbox, there is only one node, table is loaded into hive by
> hdp
> > sandbox hive script, I don't think there will have a hive consistent
> > problem
> >
> > best regards!
> >
> > --
> > View this message in context:
> >
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>

Re: Re: encouter Deserialization error when load hive table

Posted by "wangxianbin1987@gmail.com" <wa...@gmail.com>.
HDP2.3.2, kylin 2.0-rc



wangxianbin1987@gmail.com
 
From: Li Yang
Date: 2016-01-12 17:40
To: dev
Subject: Re: Re: encouter Deserialization error when load hive table
Then let's confirm your Kylin version and HDP version first. What are they?
 
On Thu, Jan 7, 2016 at 4:03 PM, xianbin wang <wa...@gmail.com>
wrote:
 
> hi all!
>
> I use HDP sandbox, there is only one node, table is loaded into hive by hdp
> sandbox hive script, I don't think there will have a hive consistent
> problem
>
> best regards!
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Re: Re: encouter Deserialization error when load hive table

Posted by Li Yang <li...@apache.org>.
Then let's confirm your Kylin version and HDP version first. What are they?

On Thu, Jan 7, 2016 at 4:03 PM, xianbin wang <wa...@gmail.com>
wrote:

> hi all!
>
> I use HDP sandbox, there is only one node, table is loaded into hive by hdp
> sandbox hive script, I don't think there will have a hive consistent
> problem
>
> best regards!
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Re: Re: encouter Deserialization error when load hive table

Posted by xianbin wang <wa...@gmail.com>.
hi all!

I use HDP sandbox, there is only one node, table is loaded into hive by hdp
sandbox hive script, I don't think there will have a hive consistent problem

best regards!

--
View this message in context: http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: Re: encouter Deserialization error when load hive table

Posted by Li Yang <li...@apache.org>.
> Task Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED

So this is an error on a MR node. Likely the hive versions are not
consistent across your cluster. A work around is let Kylin submit hive jars
as part of MR job. See https://issues.apache.org/jira/browse/KYLIN-1021


On Wed, Dec 30, 2015 at 5:16 PM, Sarnath <st...@gmail.com> wrote:

> Do you 'add jar' in hive prompt before accessing the table? If so, make
> sure that jar is  in classpath.
>

Re: Re: encouter Deserialization error when load hive table

Posted by Sarnath <st...@gmail.com>.
Do you 'add jar' in hive prompt before accessing the table? If so, make
sure that jar is  in classpath.

Re: Re: encouter Deserialization error when load hive table

Posted by "wangxianbin1987@gmail.com" <wa...@gmail.com>.
hi yang!

hive does work, any other idea?

best regards!



wangxianbin1987@gmail.com
 
From: Li Yang
Date: 2015-12-29 16:15
To: dev
Subject: Re: encouter Deserialization error when load hive table
Try hive command from the Kylin node, does simple hive queries work?
 
On Thu, Dec 24, 2015 at 2:03 PM, xianbin wang <wa...@gmail.com>
wrote:
 
> anyone have a idea for such a error,when load hive table, error as follow:
>
> L4J [2015-12-24 13:54:39,354][INFO][org.apache.hadoop.mapreduce.Job] - Task
> Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
> Error: java.io.IOException: Deserialization error: invalid stream header:
> 926A2769
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:120)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:132)
>         at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:754)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.StreamCorruptedException: invalid stream header:
> 926A2769
>         at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
>         at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         ... 11 more
>
> L4J [2015-12-24 13:54:41,105][INFO][org.apache.hadoop.mapreduce.Job] -  map
> 0% reduce 0%
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Re: encouter Deserialization error when load hive table

Posted by Li Yang <li...@apache.org>.
Try hive command from the Kylin node, does simple hive queries work?

On Thu, Dec 24, 2015 at 2:03 PM, xianbin wang <wa...@gmail.com>
wrote:

> anyone have a idea for such a error,when load hive table, error as follow:
>
> L4J [2015-12-24 13:54:39,354][INFO][org.apache.hadoop.mapreduce.Job] - Task
> Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
> Error: java.io.IOException: Deserialization error: invalid stream header:
> 926A2769
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:120)
>         at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:132)
>         at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
>         at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
>         at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:754)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.StreamCorruptedException: invalid stream header:
> 926A2769
>         at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
>         at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
>         at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
>         ... 11 more
>
> L4J [2015-12-24 13:54:41,105][INFO][org.apache.hadoop.mapreduce.Job] -  map
> 0% reduce 0%
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>