You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Steven Wong <sw...@netflix.com> on 2010/10/01 00:25:38 UTC
RE: java.lang.RuntimeException: java.lang.Double cannot be cast to
java.lang.String
I am not familiar with JsonSerde, but your data might contain doubles that you didn't know about. LOAD does not check whether the data's actual types match the types specified in the schema.
From: Hock Zoltán [mailto:hock.zoltan@gmail.com]
Sent: Thursday, September 30, 2010 8:46 AM
To: hive-user@hadoop.apache.org
Subject: java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
Hello,
I got the following exception when I made a simple query in my table.
I made the table to store email header data with the following command:
CREATE TABLE mytable (xfrom STRING, xto STRING, subject STRING, cc STRING, xgeoip STRING, receivedspf STRING, xoriginatingip STRING, messageid STRING, replyto STRING, xenvelopefrom STRING, contenttype STRING, xoriginatingip2 STRING, version STRING, fromaddr STRING, recaddr STRING) PARTITIONED BY (year int,month int,day int,dat INT) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde';
then I insert some data using LOAD DATA LOCAL INPATH "/mypipe" INTO TABLE mytable PARTITION(....); in my java app.
the pipe contains json data of course;
Everything works fine, but when I want to get some data:
hive>add jar hive-json-serde-0.1.jar;
hive>from mytable select xfrom where year=2010 month=9 day=30;
or
hive>from mytable select subject where year=2010 month=9 day=30;
these works fine.
BUT
hive>from mytable select xto where year=2010 month=9 day=30;
fails with the following error:
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
in the log:
java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:188)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String
at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:506)
at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:399)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:162)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:81)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:73)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:73)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:43)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:350)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:171)
... 4 more
I don't know where is this Double variable, everything must be stored as String.
xto field contanins the To field of the email header.
Thanks
Zoltan
RE: java.lang.RuntimeException: java.lang.Double cannot be cast to
java.lang.String
Posted by Steven Wong <sw...@netflix.com>.
One way is to inspect the data files in the table to see if there's any wrongly typed data in them. First, find the directory of your table where the files are, using Hive's "describe extended <tableName>". Then, use hadoop fs -ls to find the files and hadoop fs -cat (or -get or whatever your favorite way is) to get at the file contents.
From: Hock Zoltán [mailto:hock.zoltan@gmail.com]
Sent: Friday, October 01, 2010 1:06 AM
To: hive-user@hadoop.apache.org
Subject: Re: java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
Thank you for answering,
Is there any way to check and fix types?
Zoltan
2010/10/1 Steven Wong <sw...@netflix.com>>
I am not familiar with JsonSerde, but your data might contain doubles that you didn't know about. LOAD does not check whether the data's actual types match the types specified in the schema.
From: Hock Zoltán [mailto:hock.zoltan@gmail.com<ma...@gmail.com>]
Sent: Thursday, September 30, 2010 8:46 AM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
Hello,
I got the following exception when I made a simple query in my table.
I made the table to store email header data with the following command:
CREATE TABLE mytable (xfrom STRING, xto STRING, subject STRING, cc STRING, xgeoip STRING, receivedspf STRING, xoriginatingip STRING, messageid STRING, replyto STRING, xenvelopefrom STRING, contenttype STRING, xoriginatingip2 STRING, version STRING, fromaddr STRING, recaddr STRING) PARTITIONED BY (year int,month int,day int,dat INT) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde';
then I insert some data using LOAD DATA LOCAL INPATH "/mypipe" INTO TABLE mytable PARTITION(....); in my java app.
the pipe contains json data of course;
Everything works fine, but when I want to get some data:
hive>add jar hive-json-serde-0.1.jar;
hive>from mytable select xfrom where year=2010 month=9 day=30;
or
hive>from mytable select subject where year=2010 month=9 day=30;
these works fine.
BUT
hive>from mytable select xto where year=2010 month=9 day=30;
fails with the following error:
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.ExecDriver
in the log:
java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:188)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String
at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:506)
at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:399)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:162)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:81)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:73)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:73)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:43)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:350)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:171)
... 4 more
I don't know where is this Double variable, everything must be stored as String.
xto field contanins the To field of the email header.
Thanks
Zoltan
Re: java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
Posted by Hock Zoltán <ho...@gmail.com>.
Thank you for answering,
Is there any way to check and fix types?
Zoltan
2010/10/1 Steven Wong <sw...@netflix.com>
> I am not familiar with JsonSerde, but your data might contain doubles
> that you didn’t know about. LOAD does not check whether the data’s actual
> types match the types specified in the schema.
>
>
>
>
>
> *From:* Hock Zoltán [mailto:hock.zoltan@gmail.com]
> *Sent:* Thursday, September 30, 2010 8:46 AM
> *To:* hive-user@hadoop.apache.org
> *Subject:* java.lang.RuntimeException: java.lang.Double cannot be cast to
> java.lang.String
>
>
>
> Hello,
>
> I got the following exception when I made a simple query in my table.
> I made the table to store email header data with the following command:
>
> CREATE TABLE mytable (xfrom STRING, xto STRING, subject STRING, cc STRING,
> xgeoip STRING, receivedspf STRING, xoriginatingip STRING, messageid STRING,
> replyto STRING, xenvelopefrom STRING, contenttype STRING, xoriginatingip2
> STRING, version STRING, fromaddr STRING, recaddr STRING) PARTITIONED BY
> (year int,month int,day int,dat INT) ROW FORMAT SERDE
> 'org.apache.hadoop.hive.contrib.serde2.JsonSerde';
>
> then I insert some data using LOAD DATA LOCAL INPATH "/mypipe" INTO TABLE
> mytable PARTITION(....); in my java app.
> the pipe contains json data of course;
>
> Everything works fine, but when I want to get some data:
> hive>add jar hive-json-serde-0.1.jar;
> hive>from mytable select xfrom where year=2010 month=9 day=30;
> or
> hive>from mytable select subject where year=2010 month=9 day=30;
> these works fine.
> BUT
> hive>from mytable select xto where year=2010 month=9 day=30;
> fails with the following error:
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.ExecDriver
> in the log:
>
> java.lang.RuntimeException: java.lang.Double cannot be cast to java.lang.String
>
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:188)
>
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>
>
>
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
>
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
>
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.String
>
>
>
> at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaStringObjectInspector.getPrimitiveWritableObject(JavaStringObjectInspector.java:35)
>
> at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:506)
>
>
>
> at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:399)
>
> at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:162)
>
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
>
>
>
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
>
> at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:81)
>
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
>
>
>
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
>
> at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:73)
>
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
>
>
>
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
>
> at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:73)
>
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
>
>
>
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
>
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:43)
>
> at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:386)
>
>
>
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:598)
>
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:350)
>
> at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:171)
>
>
>
> ... 4 more
>
>
> I don't know where is this Double variable, everything must be stored as
> String.
> xto field contanins the To field of the email header.
>
>
> Thanks
> Zoltan
>