You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by vivek veeramani <vi...@gmail.com> on 2015/05/21 05:23:55 UTC

ORC Table creation failure

Hi ,

I'm trying to create a subset table from a master table which is stored in
ORC format. The query I'm using is :

 CREATE TABLE tab_20150510 STORED AS ORC AS SELECT * FROM master_tab WHERE
col_id_60 <> '2015-05-20';

The master table is a table with 60 columns with all string data type but 1
column as bigint which is not part of the query. I tried creating this
using Tez as well but no go.
The error I get is shown below:

-----
Diagnostic Messages for this Task:
Error: java.io.IOException: java.io.IOException:
java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be
cast to org.apache.hadoop.io.Text
        at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
        at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
        at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:256)
        at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:171)
        at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:198)
        at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:184)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.io.IOException: java.lang.ClassCastException:
org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text
        at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
        at
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
        at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:344)
        at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
        at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
        at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:122)
        at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:254)
        ... 11 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
cannot be cast to org.apache.hadoop.io.Text
        at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1592)
        at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1346)
        at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1788)
        at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2997)
        at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:153)
        at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:127)
        at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:339)
        ... 15 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143


FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1544   Cumulative CPU: 250716.7 sec   HDFS Read: 160494367819
HDFS Write: 121322793549 FAIL
Total MapReduce CPU Time Spent: 2 days 21 hours 38 minutes 36 seconds 700
msec


Greatly appreciate any help or information on this and how to rectify it.


-- 
Thanks ,
Vivek Veeramani


cell : +91-9632 975 975
        +91-9895 277 101

Re: ORC Table creation failure

Posted by vivek veeramani <vi...@gmail.com>.
Hi Abe,

The column in the WHERE clause is also string. I tried creating without any
filters, just a CTAS and that failed too.

Thanks,
Vivek

On Thu, May 21, 2015 at 7:47 PM, Abe Weinograd <ab...@flonet.com> wrote:

> I think your WHERE clause is having casting issues.  Is that column of the
> right data type?  Can you run the SELECT by itself without the CREATE TABLE?
>
> Abe
>
> On Wed, May 20, 2015 at 11:23 PM, vivek veeramani <
> vivek.veeramani87@gmail.com> wrote:
>
>>
>> Hi ,
>>
>> I'm trying to create a subset table from a master table which is stored
>> in ORC format. The query I'm using is :
>>
>>  CREATE TABLE tab_20150510 STORED AS ORC AS SELECT * FROM master_tab
>> WHERE col_id_60 <> '2015-05-20';
>>
>> The master table is a table with 60 columns with all string data type but
>> 1 column as bigint which is not part of the query. I tried creating this
>> using Tez as well but no go.
>> The error I get is shown below:
>>
>> -----
>> Diagnostic Messages for this Task:
>> Error: java.io.IOException: java.io.IOException:
>> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be
>> cast to org.apache.hadoop.io.Text
>>         at
>> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>>         at
>> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>>         at
>> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:256)
>>         at
>> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:171)
>>         at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:198)
>>         at
>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:184)
>>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
>>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
>>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:415)
>>         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
>> Caused by: java.io.IOException: java.lang.ClassCastException:
>> org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text
>>         at
>> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>>         at
>> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>>         at
>> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:344)
>>         at
>> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
>>         at
>> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
>>         at
>> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:122)
>>         at
>> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:254)
>>         ... 11 more
>> Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
>> cannot be cast to org.apache.hadoop.io.Text
>>         at
>> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1592)
>>         at
>> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1346)
>>         at
>> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1788)
>>         at
>> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2997)
>>         at
>> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:153)
>>         at
>> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:127)
>>         at
>> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:339)
>>         ... 15 more
>>
>> Container killed by the ApplicationMaster.
>> Container killed on request. Exit code is 143
>> Container exited with a non-zero exit code 143
>>
>>
>> FAILED: Execution Error, return code 2 from
>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>> MapReduce Jobs Launched:
>> Job 0: Map: 1544   Cumulative CPU: 250716.7 sec   HDFS Read: 160494367819
>> HDFS Write: 121322793549 FAIL
>> Total MapReduce CPU Time Spent: 2 days 21 hours 38 minutes 36 seconds 700
>> msec
>>
>>
>> Greatly appreciate any help or information on this and how to rectify it.
>>
>>
>> --
>> Thanks ,
>> Vivek Veeramani
>>
>>
>> cell : +91-9632 975 975
>>         +91-9895 277 101
>>
>
>


-- 
Thanks ,
Vivek Veeramani


cell : +91-9632 975 975
        +91-9895 277 101

Re: ORC Table creation failure

Posted by Abe Weinograd <ab...@flonet.com>.
I think your WHERE clause is having casting issues.  Is that column of the
right data type?  Can you run the SELECT by itself without the CREATE TABLE?

Abe

On Wed, May 20, 2015 at 11:23 PM, vivek veeramani <
vivek.veeramani87@gmail.com> wrote:

>
> Hi ,
>
> I'm trying to create a subset table from a master table which is stored in
> ORC format. The query I'm using is :
>
>  CREATE TABLE tab_20150510 STORED AS ORC AS SELECT * FROM master_tab WHERE
> col_id_60 <> '2015-05-20';
>
> The master table is a table with 60 columns with all string data type but
> 1 column as bigint which is not part of the query. I tried creating this
> using Tez as well but no go.
> The error I get is shown below:
>
> -----
> Diagnostic Messages for this Task:
> Error: java.io.IOException: java.io.IOException:
> java.lang.ClassCastException: org.apache.hadoop.io.IntWritable cannot be
> cast to org.apache.hadoop.io.Text
>         at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>         at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>         at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:256)
>         at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:171)
>         at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:198)
>         at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:184)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.io.IOException: java.lang.ClassCastException:
> org.apache.hadoop.io.IntWritable cannot be cast to org.apache.hadoop.io.Text
>         at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>         at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>         at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:344)
>         at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
>         at
> org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
>         at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:122)
>         at
> org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:254)
>         ... 11 more
> Caused by: java.lang.ClassCastException: org.apache.hadoop.io.IntWritable
> cannot be cast to org.apache.hadoop.io.Text
>         at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringDictionaryTreeReader.next(RecordReaderImpl.java:1592)
>         at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StringTreeReader.next(RecordReaderImpl.java:1346)
>         at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:1788)
>         at
> org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:2997)
>         at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:153)
>         at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:127)
>         at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:339)
>         ... 15 more
>
> Container killed by the ApplicationMaster.
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1544   Cumulative CPU: 250716.7 sec   HDFS Read: 160494367819
> HDFS Write: 121322793549 FAIL
> Total MapReduce CPU Time Spent: 2 days 21 hours 38 minutes 36 seconds 700
> msec
>
>
> Greatly appreciate any help or information on this and how to rectify it.
>
>
> --
> Thanks ,
> Vivek Veeramani
>
>
> cell : +91-9632 975 975
>         +91-9895 277 101
>