You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by praveenesh kumar <pr...@gmail.com> on 2012/03/23 14:05:08 UTC
Hive error: Unable to deserialize reduce input key
Hi all,
I am getting this following error when I am trying to do select ...with
group by operation.I am grouping on around 25 columns
java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error: Unable to deseralize reduce input key from
x1x128x0x0x0x0x1x254x174x1x49x55x52x46x50x53x52x46x49x46x48x0x1x142x145x93x11x1x128x87x4x73x1x128x32x107x137x1x130x165x214x131x1x49x0x1x51x48x48x120x53x48x0x1x132x11x106x192x1x128x13x178x250x1x128x0x1x0x1x78x86x0x1x55x48x50x0x1x56x57x48x53x52x0x1x50x48x54x0x1x49x51x51x55x51x0x1x48x0x1x48x46x48x0x1x48x0x1x49x55x53x55x52x54x56x55x0x1x48x0x1x0x1x0x1x0x1x0x255
...
Detailed logs...
2012-03-23 06:31:42,187 FATAL ExecReducer:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error:
Unable to deserialize reduce input key from
x1x128x0x0x0x0x0x87x66x1x54x54x46x56x55x46x48x46x48x0x1x142x124x217x207x1x128x86x17x13x1x128x29x65x57x1x130x141x82x245x1x49x0x1x49x56x48x120x49x53x48x0x1x131x235x47x199x1x128x10x161x93x1x128x0x1x0x1x73x76x0x1x55x55x51x0x1x54x48x54x50x57x0x1x56x55x0x1x49x51x51x49x48x0x1x53x0x1x56x46x48x0x1x50x50x0x1x50x52x51x57x49x52x55x56x0x1x48x0x1x0x1x0x1x0x1x0x255
with properties
{columns=_col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8,_col9,_col10,_col11,_col12,_col13,_col14,_col15,_col16,_col17,_col18,_col19,_col20,_col21,_col22,_col23,_col24,
serialization.sort.order=+++++++++++++++++++++++++,
columns.types=bigint,string,int,int,int,int,string,string,int,int,int,string,string,string,string,string,string,string,string,string,string,double,string,string,double}
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:204)
at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.serde2.SerDeException:
java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:173)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
... 7 more
Caused by: java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
at
org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:183)
at
org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:169)
... 8 more
Any help would be highly appreciated as I am kind of stuck here.
Thanks.
Re: Hive error: Unable to deserialize reduce input key
Posted by Navis류승우 <na...@nexr.com>.
I've tried to deserialize your data.
0 = bigint = -6341068275337623706
1 = string = TTFVUFHFH
2 = int = -1037822201
3 = int = -1467607277
4 = int = -1473682089
5 = int = -1337884091
6 = string = I
7 = string = IVH ISH
8 = int = -1321908327
9 = int = -1475321453
10 = int = -1476394752
11 = string = sv
12 = string = UUQ
13 = string = THTPW
14 = string = VU
15 = string = IQQIH
16 = string = S
17 = string = VFH
18 = string = PP
19 = string = PRQWIRUV
20 = string = H
21 = double = NaN
Exception in thread "main" java.io.EOFException
Could you discern columns having invalid value?
2012/9/7 praveenesh kumar <pr...@gmail.com>
> I am not sure, what can be the issue...I had it long back and got no
> response. I tried these things:
>
> 1. Increased the Child JVM heap size.
> 2. Reduced the number of reducers for the job.
> 3. Check whether your disks are not getting full while running the query.
> 3. Checked my data again. I think many times the error comes because of
> dirty data. 1 easy way to check whether data is clean or not is to count
> the number of delimiters/row. Sometimes,there are some other control
> characters instead of space that we can't see in normal text editors, use
> vi to check those also. Simple python hadoop streaming or pig scripts can
> help you to do that.
>
> Probably someone in community can give better answer to the exact problem.
>
> I hope it would help.
>
> Regards,
> Praveenesh
>
>
>
>
> On Fri, Sep 7, 2012 at 8:13 AM, 曹坤 <ca...@gmail.com> wrote:
>
>> Hi praveenesh kumar :
>> I am getting the same error today.
>> Do you have any solution ?
>>
>>
>> 2012/3/23 praveenesh kumar <pr...@gmail.com>
>>
>>> Hi all,
>>>
>>> I am getting this following error when I am trying to do select ...with
>>> group by operation.I am grouping on around 25 columns
>>>
>>> java.lang.RuntimeException:
>>> org.apache.hadoop.hive.ql.metadata.HiveException:
>>> Hive Runtime Error: Unable to deseralize reduce input key from
>>> x1x128x0x0x0x0x1x254x174x1x49x55x52x46x50x53x52x46x49x46x48x0x1x142x145x93x11x1x128x87x4x73x1x128x32x107x137x1x130x165x214x131x1x49x0x1x51x48x48x120x53x48x0x1x132x11x106x192x1x128x13x178x250x1x128x0x1x0x1x78x86x0x1x55x48x50x0x1x56x57x48x53x52x0x1x50x48x54x0x1x49x51x51x55x51x0x1x48x0x1x48x46x48x0x1x48x0x1x49x55x53x55x52x54x56x55x0x1x48x0x1x0x1x0x1x0x1x0x255
>>> ...
>>>
>>>
>>> Detailed logs...
>>>
>>> 2012-03-23 06:31:42,187 FATAL ExecReducer:
>>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error:
>>> Unable to deserialize reduce input key from
>>> x1x128x0x0x0x0x0x87x66x1x54x54x46x56x55x46x48x46x48x0x1x142x124x217x207x1x128x86x17x13x1x128x29x65x57x1x130x141x82x245x1x49x0x1x49x56x48x120x49x53x48x0x1x131x235x47x199x1x128x10x161x93x1x128x0x1x0x1x73x76x0x1x55x55x51x0x1x54x48x54x50x57x0x1x56x55x0x1x49x51x51x49x48x0x1x53x0x1x56x46x48x0x1x50x50x0x1x50x52x51x57x49x52x55x56x0x1x48x0x1x0x1x0x1x0x1x0x255
>>> with properties
>>> {columns=_col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8,_col9,_col10,_col11,_col12,_col13,_col14,_col15,_col16,_col17,_col18,_col19,_col20,_col21,_col22,_col23,_col24,
>>> serialization.sort.order=+++++++++++++++++++++++++,
>>> columns.types=bigint,string,int,int,int,int,string,string,int,int,int,string,string,string,string,string,string,string,string,string,string,double,string,string,double}
>>> at
>>> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:204)
>>> at
>>> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
>>> at
>>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>> at java.security.AccessController.doPrivileged(Native
>>> Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
>>> java.io.EOFException
>>> at
>>> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:173)
>>> at
>>> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
>>> ... 7 more
>>> Caused by: java.io.EOFException
>>> at
>>> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
>>> at
>>> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:183)
>>> at
>>> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:169)
>>> ... 8 more
>>>
>>>
>>> Any help would be highly appreciated as I am kind of stuck here.
>>>
>>> Thanks.
>>>
>>
>>
>>
>> --
>> Best wishs！
>> 曹 坤
>>
>>
>
Re: Hive error: Unable to deserialize reduce input key
Posted by praveenesh kumar <pr...@gmail.com>.
I am not sure, what can be the issue...I had it long back and got no
response. I tried these things:
1. Increased the Child JVM heap size.
2. Reduced the number of reducers for the job.
3. Check whether your disks are not getting full while running the query.
3. Checked my data again. I think many times the error comes because of
dirty data. 1 easy way to check whether data is clean or not is to count
the number of delimiters/row. Sometimes,there are some other control
characters instead of space that we can't see in normal text editors, use
vi to check those also. Simple python hadoop streaming or pig scripts can
help you to do that.
Probably someone in community can give better answer to the exact problem.
I hope it would help.
Regards,
Praveenesh
On Fri, Sep 7, 2012 at 8:13 AM, 曹坤 <ca...@gmail.com> wrote:
> Hi praveenesh kumar :
> I am getting the same error today.
> Do you have any solution ?
>
>
> 2012/3/23 praveenesh kumar <pr...@gmail.com>
>
>> Hi all,
>>
>> I am getting this following error when I am trying to do select ...with
>> group by operation.I am grouping on around 25 columns
>>
>> java.lang.RuntimeException:
>> org.apache.hadoop.hive.ql.metadata.HiveException:
>> Hive Runtime Error: Unable to deseralize reduce input key from
>> x1x128x0x0x0x0x1x254x174x1x49x55x52x46x50x53x52x46x49x46x48x0x1x142x145x93x11x1x128x87x4x73x1x128x32x107x137x1x130x165x214x131x1x49x0x1x51x48x48x120x53x48x0x1x132x11x106x192x1x128x13x178x250x1x128x0x1x0x1x78x86x0x1x55x48x50x0x1x56x57x48x53x52x0x1x50x48x54x0x1x49x51x51x55x51x0x1x48x0x1x48x46x48x0x1x48x0x1x49x55x53x55x52x54x56x55x0x1x48x0x1x0x1x0x1x0x1x0x255
>> ...
>>
>>
>> Detailed logs...
>>
>> 2012-03-23 06:31:42,187 FATAL ExecReducer:
>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error:
>> Unable to deserialize reduce input key from
>> x1x128x0x0x0x0x0x87x66x1x54x54x46x56x55x46x48x46x48x0x1x142x124x217x207x1x128x86x17x13x1x128x29x65x57x1x130x141x82x245x1x49x0x1x49x56x48x120x49x53x48x0x1x131x235x47x199x1x128x10x161x93x1x128x0x1x0x1x73x76x0x1x55x55x51x0x1x54x48x54x50x57x0x1x56x55x0x1x49x51x51x49x48x0x1x53x0x1x56x46x48x0x1x50x50x0x1x50x52x51x57x49x52x55x56x0x1x48x0x1x0x1x0x1x0x1x0x255
>> with properties
>> {columns=_col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8,_col9,_col10,_col11,_col12,_col13,_col14,_col15,_col16,_col17,_col18,_col19,_col20,_col21,_col22,_col23,_col24,
>> serialization.sort.order=+++++++++++++++++++++++++,
>> columns.types=bigint,string,int,int,int,int,string,string,int,int,int,string,string,string,string,string,string,string,string,string,string,double,string,string,double}
>> at
>> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:204)
>> at
>> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
>> at
>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native
>> Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
>> java.io.EOFException
>> at
>> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:173)
>> at
>> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
>> ... 7 more
>> Caused by: java.io.EOFException
>> at
>> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
>> at
>> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:183)
>> at
>> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:169)
>> ... 8 more
>>
>>
>> Any help would be highly appreciated as I am kind of stuck here.
>>
>> Thanks.
>>
>
>
>
> --
> Best wishs！
> 曹 坤
>
>
Re: Hive error: Unable to deserialize reduce input key
Posted by 曹坤 <ca...@gmail.com>.
Hi praveenesh kumar :
I am getting the same error today.
Do you have any solution ?
2012/3/23 praveenesh kumar <pr...@gmail.com>
> Hi all,
>
> I am getting this following error when I am trying to do select ...with
> group by operation.I am grouping on around 25 columns
>
> java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error: Unable to deseralize reduce input key from
> x1x128x0x0x0x0x1x254x174x1x49x55x52x46x50x53x52x46x49x46x48x0x1x142x145x93x11x1x128x87x4x73x1x128x32x107x137x1x130x165x214x131x1x49x0x1x51x48x48x120x53x48x0x1x132x11x106x192x1x128x13x178x250x1x128x0x1x0x1x78x86x0x1x55x48x50x0x1x56x57x48x53x52x0x1x50x48x54x0x1x49x51x51x55x51x0x1x48x0x1x48x46x48x0x1x48x0x1x49x55x53x55x52x54x56x55x0x1x48x0x1x0x1x0x1x0x1x0x255
> ...
>
>
> Detailed logs...
>
> 2012-03-23 06:31:42,187 FATAL ExecReducer:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error:
> Unable to deserialize reduce input key from
> x1x128x0x0x0x0x0x87x66x1x54x54x46x56x55x46x48x46x48x0x1x142x124x217x207x1x128x86x17x13x1x128x29x65x57x1x130x141x82x245x1x49x0x1x49x56x48x120x49x53x48x0x1x131x235x47x199x1x128x10x161x93x1x128x0x1x0x1x73x76x0x1x55x55x51x0x1x54x48x54x50x57x0x1x56x55x0x1x49x51x51x49x48x0x1x53x0x1x56x46x48x0x1x50x50x0x1x50x52x51x57x49x52x55x56x0x1x48x0x1x0x1x0x1x0x1x0x255
> with properties
> {columns=_col0,_col1,_col2,_col3,_col4,_col5,_col6,_col7,_col8,_col9,_col10,_col11,_col12,_col13,_col14,_col15,_col16,_col17,_col18,_col19,_col20,_col21,_col22,_col23,_col24,
> serialization.sort.order=+++++++++++++++++++++++++,
> columns.types=bigint,string,int,int,int,int,string,string,int,int,int,string,string,string,string,string,string,string,string,string,string,double,string,string,double}
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:204)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
> at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.serde2.SerDeException:
> java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:173)
> at
> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
> ... 7 more
> Caused by: java.io.EOFException
> at
> org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
> at
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:183)
> at
> org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.deserialize(BinarySortableSerDe.java:169)
> ... 8 more
>
>
> Any help would be highly appreciated as I am kind of stuck here.
>
> Thanks.
>
--
Best wishs！
曹 坤