[jira] [Resolved] (SPARK-26535) Parsing literals as DOUBLE instead of DECIMAL

["Dongjoon Hyun (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-26535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun resolved SPARK-26535.
-----------------------------------
    Resolution: Won't Do

Hi, [~mgaido]. 

First of all, Hive starts to use `Decimal` by default. Also, this introduces TPCDS-H query result difference between Spark versions. We cannot do this.

{code}
hive> select version();
OK
3.1.1 rf4e0529634b6231a0072295da48af466cf2f10b7
Time taken: 0.089 seconds, Fetched: 1 row(s)
hive> explain select 2.3;
OK
STAGE DEPENDENCIES:
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        TableScan
          alias: _dummy_table
          Row Limit Per Split: 1
          Statistics: Num rows: 1 Data size: 10 Basic stats: COMPLETE Column stats: COMPLETE
          Select Operator
            expressions: 2.3 (type: decimal(2,1))
            outputColumnNames: _col0
            Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
            ListSink
{code}

> Parsing literals as DOUBLE instead of DECIMAL
> ---------------------------------------------
>
>                 Key: SPARK-26535
>                 URL: https://issues.apache.org/jira/browse/SPARK-26535
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Marco Gaido
>            Priority: Major
>
> As pointed out by [~dkbiswal]'s comment https://github.com/apache/spark/pull/22450#issuecomment-423082389, most of other RDBMS (DB2, Presto, Hive, MSSQL) consider literals as DOUBLE by default.
> Spark as of now consider them as DECIMAL. This is quite problematic especially in relation with the operations on decimal, for which we base our implementation on Hive/MSSQL.
> So this ticket is for moving by default the resolution of literals as DOUBLE, but with a config which allows to get back to the previous behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org