You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by xianbin wang <wa...@gmail.com> on 2015/12/24 07:03:22 UTC
encouter Deserialization error when load hive table
anyone have a idea for such a error,when load hive table, error as follow:
L4J [2015-12-24 13:54:39,354][INFO][org.apache.hadoop.mapreduce.Job] - Task
Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
Error: java.io.IOException: Deserialization error: invalid stream header:
926A2769
at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:120)
at
org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:132)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:754)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.StreamCorruptedException: invalid stream header: 926A2769
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
... 11 more
L4J [2015-12-24 13:54:41,105][INFO][org.apache.hadoop.mapreduce.Job] - map
0% reduce 0%
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Re: encouter Deserialization error when load hive table
Posted by xianbin wang <wa...@gmail.com>.
hi yang!
sorry for reply so late, how could hbase be involved in this stage, it seem
like MR job use HCat load data from hive table, it is not reasonable.
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3441.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Re: encouter Deserialization error when load hive table
Posted by Li Yang <li...@apache.org>.
Kylin 2.0-rc still requires hbase 0.9x, it does not work with HBase 1.x.
That likely be the problem.
On Tue, Jan 12, 2016 at 5:55 PM, wangxianbin1987@gmail.com <
wangxianbin1987@gmail.com> wrote:
> HDP2.3.2, kylin 2.0-rc
>
>
>
> wangxianbin1987@gmail.com
>
> From: Li Yang
> Date: 2016-01-12 17:40
> To: dev
> Subject: Re: Re: encouter Deserialization error when load hive table
> Then let's confirm your Kylin version and HDP version first. What are they?
>
> On Thu, Jan 7, 2016 at 4:03 PM, xianbin wang <wa...@gmail.com>
> wrote:
>
> > hi all!
> >
> > I use HDP sandbox, there is only one node, table is loaded into hive by
> hdp
> > sandbox hive script, I don't think there will have a hive consistent
> > problem
> >
> > best regards!
> >
> > --
> > View this message in context:
> >
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
Re: Re: encouter Deserialization error when load hive table
Posted by "wangxianbin1987@gmail.com" <wa...@gmail.com>.
HDP2.3.2, kylin 2.0-rc
wangxianbin1987@gmail.com
From: Li Yang
Date: 2016-01-12 17:40
To: dev
Subject: Re: Re: encouter Deserialization error when load hive table
Then let's confirm your Kylin version and HDP version first. What are they?
On Thu, Jan 7, 2016 at 4:03 PM, xianbin wang <wa...@gmail.com>
wrote:
> hi all!
>
> I use HDP sandbox, there is only one node, table is loaded into hive by hdp
> sandbox hive script, I don't think there will have a hive consistent
> problem
>
> best regards!
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
Re: Re: encouter Deserialization error when load hive table
Posted by Li Yang <li...@apache.org>.
Then let's confirm your Kylin version and HDP version first. What are they?
On Thu, Jan 7, 2016 at 4:03 PM, xianbin wang <wa...@gmail.com>
wrote:
> hi all!
>
> I use HDP sandbox, there is only one node, table is loaded into hive by hdp
> sandbox hive script, I don't think there will have a hive consistent
> problem
>
> best regards!
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
Re: Re: encouter Deserialization error when load hive table
Posted by xianbin wang <wa...@gmail.com>.
hi all!
I use HDP sandbox, there is only one node, table is loaded into hive by hdp
sandbox hive script, I don't think there will have a hive consistent problem
best regards!
--
View this message in context: http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888p3118.html
Sent from the Apache Kylin mailing list archive at Nabble.com.
Re: Re: encouter Deserialization error when load hive table
Posted by Li Yang <li...@apache.org>.
> Task Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
So this is an error on a MR node. Likely the hive versions are not
consistent across your cluster. A work around is let Kylin submit hive jars
as part of MR job. See https://issues.apache.org/jira/browse/KYLIN-1021
On Wed, Dec 30, 2015 at 5:16 PM, Sarnath <st...@gmail.com> wrote:
> Do you 'add jar' in hive prompt before accessing the table? If so, make
> sure that jar is in classpath.
>
Re: Re: encouter Deserialization error when load hive table
Posted by Sarnath <st...@gmail.com>.
Do you 'add jar' in hive prompt before accessing the table? If so, make
sure that jar is in classpath.
Re: Re: encouter Deserialization error when load hive table
Posted by "wangxianbin1987@gmail.com" <wa...@gmail.com>.
hi yang!
hive does work, any other idea?
best regards!
wangxianbin1987@gmail.com
From: Li Yang
Date: 2015-12-29 16:15
To: dev
Subject: Re: encouter Deserialization error when load hive table
Try hive command from the Kylin node, does simple hive queries work?
On Thu, Dec 24, 2015 at 2:03 PM, xianbin wang <wa...@gmail.com>
wrote:
> anyone have a idea for such a error,when load hive table, error as follow:
>
> L4J [2015-12-24 13:54:39,354][INFO][org.apache.hadoop.mapreduce.Job] - Task
> Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
> Error: java.io.IOException: Deserialization error: invalid stream header:
> 926A2769
> at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:120)
> at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:132)
> at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
> at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
> at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:754)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.StreamCorruptedException: invalid stream header:
> 926A2769
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
> at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
> ... 11 more
>
> L4J [2015-12-24 13:54:41,105][INFO][org.apache.hadoop.mapreduce.Job] - map
> 0% reduce 0%
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
Re: encouter Deserialization error when load hive table
Posted by Li Yang <li...@apache.org>.
Try hive command from the Kylin node, does simple hive queries work?
On Thu, Dec 24, 2015 at 2:03 PM, xianbin wang <wa...@gmail.com>
wrote:
> anyone have a idea for such a error,when load hive table, error as follow:
>
> L4J [2015-12-24 13:54:39,354][INFO][org.apache.hadoop.mapreduce.Job] - Task
> Id : attempt_1450856278246_0003_m_000000_0, Status : FAILED
> Error: java.io.IOException: Deserialization error: invalid stream header:
> 926A2769
> at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:120)
> at
> org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:132)
> at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
> at
>
> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
> at
> org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:372)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:754)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.io.StreamCorruptedException: invalid stream header:
> 926A2769
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
> at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
> at
> org.apache.hive.hcatalog.common.HCatUtil.deserialize(HCatUtil.java:117)
> ... 11 more
>
> L4J [2015-12-24 13:54:41,105][INFO][org.apache.hadoop.mapreduce.Job] - map
> 0% reduce 0%
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/encouter-Deserialization-error-when-load-hive-table-tp2888.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>