You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2019/05/10 04:52:00 UTC
[jira] [Commented] (SPARK-26437) Decimal data becomes bigint to
query, unable to query
[ https://issues.apache.org/jira/browse/SPARK-26437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836878#comment-16836878 ]
Xiao Li commented on SPARK-26437:
---------------------------------
Even if we do not use our native ORC reader, Spark 3.0 will be able to read it when enabling Hadoop 3.2 profile since we upgrade Hive executive JAR from 1.2.1 too 2.3.4. See the PR https://github.com/apache/spark/pull/24391
> Decimal data becomes bigint to query, unable to query
> -----------------------------------------------------
>
> Key: SPARK-26437
> URL: https://issues.apache.org/jira/browse/SPARK-26437
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.3, 2.0.2, 2.1.3, 2.2.2, 2.3.1
> Reporter: zengxl
> Priority: Major
> Fix For: 3.0.0
>
>
> this is my sql:
> create table tmp.tmp_test_6387_1224_spark stored as ORCFile as select 0.00 as a
> select a from tmp.tmp_test_6387_1224_spark
> CREATE TABLE `tmp.tmp_test_6387_1224_spark`(
> {color:#f79232} `a` decimal(2,2)){color}
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> When I query this table（use hive or sparksql，the exception is same）, I throw the following exception information
> *Caused by: java.io.EOFException: Reading BigInteger past EOF from compressed stream Stream for column 1 kind DATA position: 0 length: 0 range: 0 offset: 0 limit: 0*
> *at org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readBigInteger(SerializationUtils.java:176)*
> *at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$DecimalTreeReader.next(TreeReaderFactory.java:1264)*
> *at org.apache.hadoop.hive.ql.io.orc.TreeReaderFactory$StructTreeReader.next(TreeReaderFactory.java:2004)*
> *at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1039)*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org