You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Eva Tse <et...@netflix.com> on 2009/08/31 06:03:29 UTC
Array index out of bounds exception ...
We run the following query:
> create table test_map (other_properties map<string, string>);
> insert overwrite table test_map
> select other_properties from log_table where dateint = 20090825;
The other_properties column in log_table is also of the same type. We are
using such a query to merge small files in our tables.
We get the following array index out of bounds exceptions from all the
reducers of the 2nd map-reduce job and the query failed.
java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 308
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:465)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 308
at org.apache.hadoop.hive.serde2.lazy.LazyMap.parse(LazyMap.java:172)
at org.apache.hadoop.hive.serde2.lazy.LazyMap.getMap(LazyMap.java:293)
at
org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector.ge
tMap(LazyMapObjectInspector.java:100)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe
.java:453)
at
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe
.java:385)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.jav
a:151)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:464)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.process(ExtractOperator.java:
42)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:193)
... 3 more
Is this a known issue? Our hive version is trunk version 795559 running on
hadoop 0.20.
Thanks a lot!
Eva.
Re: Array index out of bounds exception ...
Posted by Eva Tse <et...@netflix.com>.
Thanks for pointing that out.
On 8/30/09 11:37 PM, "Zheng Shao" <zs...@gmail.com> wrote:
> Yes, see https://issues.apache.org/jira/browse/HIVE-719
>
> Zheng
>
> On Sun, Aug 30, 2009 at 9:03 PM, Eva Tse <et...@netflix.com> wrote:
>> We run the following query:
>>
>>> create table test_map (other_properties map<string, string>);
>>> insert overwrite table test_map
>>> select other_properties from log_table where dateint = 20090825;
>>
>> The other_properties column in log_table is also of the same type. We are
>> using such a query to merge small files in our tables.
>>
>> We get the following array index out of bounds exceptions from all the
>> reducers of the 2nd map-reduce job and the query failed.
>>
>> java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 308
>> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
>> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:465)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 308
>> at org.apache.hadoop.hive.serde2.lazy.LazyMap.parse(LazyMap.java:172)
>> at org.apache.hadoop.hive.serde2.lazy.LazyMap.getMap(LazyMap.java:293)
>> at
>> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector.get
>> Map(LazyMapObjectInspector.java:100)
>> at
>> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.
>> java:453)
>> at
>> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.
>> java:385)
>> at
>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java
>> :151)
>> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:464)
>> at
>> org.apache.hadoop.hive.ql.exec.ExtractOperator.process(ExtractOperator.java:4
>> 2)
>> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:193)
>> ... 3 more
>>
>>
>> Is this a known issue? Our hive version is trunk version 795559 running on
>> hadoop 0.20.
>>
>> Thanks a lot!
>> Eva.
>
>
Re: Array index out of bounds exception ...
Posted by Zheng Shao <zs...@gmail.com>.
Yes, see https://issues.apache.org/jira/browse/HIVE-719
Zheng
On Sun, Aug 30, 2009 at 9:03 PM, Eva Tse <et...@netflix.com> wrote:
> We run the following query:
>
> create table test_map (other_properties map<string, string>);
> insert overwrite table test_map
> select other_properties from log_table where dateint = 20090825;
>
>
> The other_properties column in log_table is also of the same type. We are
> using such a query to merge small files in our tables.
>
> We get the following array index out of bounds exceptions from all the
> reducers of the 2nd map-reduce job and the query failed.
>
> java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: 308
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:202)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:465)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:413)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 308
> at org.apache.hadoop.hive.serde2.lazy.LazyMap.parse(LazyMap.java:172)
> at org.apache.hadoop.hive.serde2.lazy.LazyMap.getMap(LazyMap.java:293)
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector.getMap(LazyMapObjectInspector.java:100)
> at
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:453)
> at
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:385)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:151)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:464)
> at
> org.apache.hadoop.hive.ql.exec.ExtractOperator.process(ExtractOperator.java:42)
> at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:193)
> ... 3 more
>
>
> Is this a known issue? Our hive version is trunk version *795559 *running
> on hadoop 0.20.
>
> Thanks a lot!
> Eva.
>
--
Yours,
Zheng