You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2018/10/19 16:43:00 UTC
[jira] [Comment Edited] (SPARK-25413) [2.3.2.rc5 Blocker] Precision
Value is going for toss when Avg is done
[ https://issues.apache.org/jira/browse/SPARK-25413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657058#comment-16657058 ]
Dongjoon Hyun edited comment on SPARK-25413 at 10/19/18 4:42 PM:
-----------------------------------------------------------------
Thank you for reporting and making a PR, [~sandeep.katta2007] and [~ajithshetty].
According to the discussion on the JIRA issue and the given example is controlled by configuration, I'll close this as `Invalid` issue.
{code}
scala> sql("set spark.sql.decimalOperations.allowPrecisionLoss=false")
scala> sql("select avg(salary)+10 from hiveBigDecimal").show(false)
+----------------------------------------------------------------------------------------+
|(CAST(avg(salary) AS DECIMAL(32,14)) + CAST(CAST(10 AS DECIMAL(2,0)) AS DECIMAL(32,14)))|
+----------------------------------------------------------------------------------------+
|37800224355780013.75982042536364 |
+----------------------------------------------------------------------------------------+
{code}
was (Author: dongjoon):
Thank you for reporting and making a PR, [~sandeep.katta2007].
According to the discussion on the JIRA issue and the given example is controlled by configuration, I'll close this as `Invalid` issue.
{code}
scala> sql("set spark.sql.decimalOperations.allowPrecisionLoss=false")
scala> sql("select avg(salary)+10 from hiveBigDecimal").show(false)
+----------------------------------------------------------------------------------------+
|(CAST(avg(salary) AS DECIMAL(32,14)) + CAST(CAST(10 AS DECIMAL(2,0)) AS DECIMAL(32,14)))|
+----------------------------------------------------------------------------------------+
|37800224355780013.75982042536364 |
+----------------------------------------------------------------------------------------+
{code}
> [2.3.2.rc5 Blocker] Precision Value is going for toss when Avg is done
> ----------------------------------------------------------------------
>
> Key: SPARK-25413
> URL: https://issues.apache.org/jira/browse/SPARK-25413
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.1
> Environment: Csv FIle content
>
> 1,23-07-2015,India,aaa1,phone197,ASD69643,1.23457E+16
> 2,24-07-2015,India,aaa2,phone756,ASD42892,1.23457E+16
> 3,25-07-2015,India,aaa3,phone1904,ASD37014,1.23457E+16
> 4,26-07-2015,India,aaa4,phone2435,ASD66902,1.23457E+16
> 5,27-07-2015,India,aaa5,phone2441,ASD90633,2.23457E+16
> 6,28-07-2015,India,aaa6,phone294,ASD59961,3.23457E+16
> 7,29-07-2015,India,aaa7,phone610,ASD14875,4.23457E+16
> 8,30-07-2015,India,aaa8,phone1848,ASD57308,5.23457E+16
> 9,18-07-2015,India,aaa9,phone706,ASD86717,6.23457E+16
> 10,19-07-2015,usa,aaa10,phone685,ASD30505,7.23457E+16
> 11,18-07-2015,china,aaa11,phone1554,ASD26101,8.23457E+16
> Reporter: sandeep katta
> Priority: Blocker
> Attachments: decimalBoundaryDataHive.csv
>
>
> sql("create table if not exists hiveBigDecimal(ID Int, date Timestamp, country String, name String, phonetype String, serialname String, salary decimal(27, 10))row format delimited fields terminated by ','")
> sql(s"LOAD DATA local inpath '$resourcesPath/decimalBoundaryDataHive.csv' INTO table hiveBigDecimal")
> sql("select avg(salary)+10 from hiveBigDecimal").show(false)
>
> Output with 2.3.1
> +----------------------------------------------------------------------------------------+
> |(CAST(avg(salary) AS DECIMAL(32,14)) + CAST(CAST(10 AS DECIMAL(2,0)) AS DECIMAL(32,14)))|
> +----------------------------------------------------------------------------------------+
> |37800224355780013.75982042536364|
> +----------------------------------------------------------------------------------------+
> OutPut with 2.3.2_RC5
> |(CAST(avg(salary) AS DECIMAL(32,14)) + CAST(CAST(10 AS DECIMAL(2,0)) AS DECIMAL(32,14)))|
> +----------------------------------------------------------------------------------------+
> |37800224355780013.75982042536000 |
> +----------------------------------------------------------------------------------------
> *PS:If I revert SPARK-24957 then 2.3.1 and 2.3.2_rc5 output is same*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org