You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "aloyszhang (JIRA)" <ji...@apache.org> on 2019/07/08 03:42:00 UTC
[jira] [Updated] (HIVE-21964) jdbc handler class cast exception
[ https://issues.apache.org/jira/browse/HIVE-21964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
aloyszhang updated HIVE-21964:
------------------------------
Description:
Using hive jdbc handler to query external mysql data source with type decimal type, it throws class cast Exception :
{code:java}
2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main] CliDriver: Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
... 14 more
Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveJavaObject(JavaHiveDecimalObjectInspector.java:55)
at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:329)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:292)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:247)
at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:72)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:231)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
at org.apache.hadoop.hive.serde2.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:67)
at org.apache.hadoop.hive.serde2.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:36)
at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
... 24 more
{code}
Same problem with type date and timestamp.
I debug the code and find that was caused by hive-jdbc-handler return result wity type
java.math.Decimal , but when serialize the result row by calling JavaHiveDecimalObjectInspector#getPrimitiveJavaObject , java.math.Decimal cann't cast to org.apache.hadoop.hive.common.type.HiveDecimal.
So, IMO, this is a bug in jdbc-handler code when processing decimal, date and timestamp teyps. I think jdbc-handler should return result with type cast like this :
{code:java}
@Override
public Map<String, Object> next() {
try {
ResultSetMetaData metadata = rs.getMetaData();
int numColumns = metadata.getColumnCount();
Map<String, Object> record = new HashMap<String, Object>(numColumns);
for (int i = 0; i < numColumns; i++) {
String key = metadata.getColumnName(i + 1);
Object value;
if (columnTypes!=null && columnTypes.get(i) instanceof PrimitiveTypeInfo) {
// This is not a complete list, barely make information schema work
switch (((PrimitiveTypeInfo)columnTypes.get(i)).getTypeName()) {
case "int":
case "smallint":
case "tinyint":
value = rs.getInt(i + 1);
break;
case "bigint":
value = rs.getLong(i + 1);
break;
case "float":
value = rs.getFloat(i + 1);
break;
case "double":
value = rs.getDouble(i + 1);
break;
case "decimal":
case "bigdecimal":
value = HiveDecimal.create(rs.getBigDecimal(i + 1));
break;
case "boolean":
value = rs.getBoolean(i + 1);
break;
case "string":
case "char":
case "varchar":
value = rs.getString(i + 1);
break;
case "date":
case "datetime":
value = new Date(rs.getDate(i + 1).toLocalDate());
break;
case "timestamp":
value = new Timestamp((rs.getTimestamp(i + 1)).toLocalDateTime());
break;
default:
value = rs.getObject(i + 1);
break;
}
} else {
value = rs.getObject(i + 1);
}
record.put(key, value);
}
return record;
}
catch (Exception e) {
LOGGER.warn("next() threw exception", e);
return null;
}
}
{code}
As I am quite a novice, So I want to know whether this change has other effects or not .
Any suggestions are appreciated.
was:
Using hive jdbc handler to query external mysql data source with type decimal type, it throws class cast Exception :
{code:java}
2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main] CliDriver: Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
... 14 more
Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveJavaObject(JavaHiveDecimalObjectInspector.java:55)
at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:329)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:292)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:247)
at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:72)
at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:231)
at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
at org.apache.hadoop.hive.serde2.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:67)
at org.apache.hadoop.hive.serde2.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:36)
at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
... 24 more
{code}
Some problem with type date and timestamp.
I debug the code and find that was caused by hive-jdbc-handler return result wity type
java.math.Decimal , but when serialize the result row by calling JavaHiveDecimalObjectInspector#getPrimitiveJavaObject , java.math.Decimal cann't cast to org.apache.hadoop.hive.common.type.HiveDecimal.
So, IMO, this is a bug in jdbc-handler code when processing decimal, date and timestamp teyps. I think jdbc-handler should return result with type cast like this :
{code:java}
@Override
public Map<String, Object> next() {
try {
ResultSetMetaData metadata = rs.getMetaData();
int numColumns = metadata.getColumnCount();
Map<String, Object> record = new HashMap<String, Object>(numColumns);
for (int i = 0; i < numColumns; i++) {
String key = metadata.getColumnName(i + 1);
Object value;
if (columnTypes!=null && columnTypes.get(i) instanceof PrimitiveTypeInfo) {
// This is not a complete list, barely make information schema work
switch (((PrimitiveTypeInfo)columnTypes.get(i)).getTypeName()) {
case "int":
case "smallint":
case "tinyint":
value = rs.getInt(i + 1);
break;
case "bigint":
value = rs.getLong(i + 1);
break;
case "float":
value = rs.getFloat(i + 1);
break;
case "double":
value = rs.getDouble(i + 1);
break;
case "decimal":
case "bigdecimal":
value = HiveDecimal.create(rs.getBigDecimal(i + 1));
break;
case "boolean":
value = rs.getBoolean(i + 1);
break;
case "string":
case "char":
case "varchar":
value = rs.getString(i + 1);
break;
case "date":
case "datetime":
value = new Date(rs.getDate(i + 1).toLocalDate());
break;
case "timestamp":
value = new Timestamp((rs.getTimestamp(i + 1)).toLocalDateTime());
break;
default:
value = rs.getObject(i + 1);
break;
}
} else {
value = rs.getObject(i + 1);
}
record.put(key, value);
}
return record;
}
catch (Exception e) {
LOGGER.warn("next() threw exception", e);
return null;
}
}
{code}
As I am quite a novice, So I want to know whether this change has other effects or not .
Any suggestions are appreciated.
> jdbc handler class cast exception
> ---------------------------------
>
> Key: HIVE-21964
> URL: https://issues.apache.org/jira/browse/HIVE-21964
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
> Affects Versions: 3.1.1
> Reporter: aloyszhang
> Priority: Major
>
> Using hive jdbc handler to query external mysql data source with type decimal type, it throws class cast Exception :
>
> {code:java}
> 2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main] CliDriver: Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ... 14 more
> Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast to org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveJavaObject(JavaHiveDecimalObjectInspector.java:55)
> at org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:329)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:292)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:247)
> at org.apache.hadoop.hive.serde2.DelimitedJSONSerDe.serializeField(DelimitedJSONSerDe.java:72)
> at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.doSerialize(LazySimpleSerDe.java:231)
> at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.serialize(AbstractEncodingAwareSerDe.java:55)
> at org.apache.hadoop.hive.serde2.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:67)
> at org.apache.hadoop.hive.serde2.DefaultFetchFormatter.convert(DefaultFetchFormatter.java:36)
> at org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:94)
> ... 24 more
> {code}
> Same problem with type date and timestamp.
>
> I debug the code and find that was caused by hive-jdbc-handler return result wity type
> java.math.Decimal , but when serialize the result row by calling JavaHiveDecimalObjectInspector#getPrimitiveJavaObject , java.math.Decimal cann't cast to org.apache.hadoop.hive.common.type.HiveDecimal.
> So, IMO, this is a bug in jdbc-handler code when processing decimal, date and timestamp teyps. I think jdbc-handler should return result with type cast like this :
> {code:java}
> @Override
> public Map<String, Object> next() {
> try {
> ResultSetMetaData metadata = rs.getMetaData();
> int numColumns = metadata.getColumnCount();
> Map<String, Object> record = new HashMap<String, Object>(numColumns);
> for (int i = 0; i < numColumns; i++) {
> String key = metadata.getColumnName(i + 1);
> Object value;
> if (columnTypes!=null && columnTypes.get(i) instanceof PrimitiveTypeInfo) {
> // This is not a complete list, barely make information schema work
> switch (((PrimitiveTypeInfo)columnTypes.get(i)).getTypeName()) {
> case "int":
> case "smallint":
> case "tinyint":
> value = rs.getInt(i + 1);
> break;
> case "bigint":
> value = rs.getLong(i + 1);
> break;
> case "float":
> value = rs.getFloat(i + 1);
> break;
> case "double":
> value = rs.getDouble(i + 1);
> break;
> case "decimal":
> case "bigdecimal":
> value = HiveDecimal.create(rs.getBigDecimal(i + 1));
> break;
> case "boolean":
> value = rs.getBoolean(i + 1);
> break;
> case "string":
> case "char":
> case "varchar":
> value = rs.getString(i + 1);
> break;
> case "date":
> case "datetime":
> value = new Date(rs.getDate(i + 1).toLocalDate());
> break;
> case "timestamp":
> value = new Timestamp((rs.getTimestamp(i + 1)).toLocalDateTime());
> break;
> default:
> value = rs.getObject(i + 1);
> break;
> }
> } else {
> value = rs.getObject(i + 1);
> }
> record.put(key, value);
> }
> return record;
> }
> catch (Exception e) {
> LOGGER.warn("next() threw exception", e);
> return null;
> }
> }
> {code}
> As I am quite a novice, So I want to know whether this change has other effects or not .
> Any suggestions are appreciated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)