You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:00:25 UTC
[jira] [Updated] (SPARK-19102) Accuracy error of spark SQL results
[ https://issues.apache.org/jira/browse/SPARK-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-19102:
---------------------------------
Labels: bulk-closed (was: )
> Accuracy error of spark SQL results
> -----------------------------------
>
> Key: SPARK-19102
> URL: https://issues.apache.org/jira/browse/SPARK-19102
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0, 1.6.1
> Environment: Spark 1.6.0, Hadoop 2.6.0,JDK 1.8,CentOS6.6
> Reporter: XiaodongCui
> Priority: Major
> Labels: bulk-closed
> Attachments: a.zip
>
>
> the problem is cube6's second column named sumprice is 10000 times bigger than the cube5's second column named sumprice,but they should be equal .the first sql is "SELECT areacode1, SUM(quantity*unitprice) AS sumprice FROM hd_salesflat GROUP BY areacode1" , the second sql is "SELECT areacode1, SUM(quantity*unitprice) AS sumprice, COUNT(DISTINCT transno) FROM hd_salesflat GROUP BY areacode1",the result of sumprice should be equal,but actually they are not ,that's the problem.
> code:
> ================================================================================
> DataFrame df1=sqlContext.read().parquet("hdfs://cdh01:8020/sandboxdata_A/test/a");
> df1.registerTempTable("hd_salesflat");
> DataFrame cube5 = sqlContext.sql("SELECT areacode1, SUM(quantity*unitprice) AS sumprice FROM hd_salesflat GROUP BY areacode1");
> DataFrame cube6 = sqlContext.sql("SELECT areacode1, SUM(quantity*unitprice) AS sumprice, COUNT(DISTINCT transno) FROM hd_salesflat GROUP BY areacode1");
> cube5.select("sumprice").show(50);
> cube6.select("sumprice").show(50);
> ================================================================================
> my data has only one row and four column in the attach file：
> transno | quantity | unitprice | areacode1
> 76317828| 1.0000 | 25.0000 | HDCN
> data schema：
> |-- areacode1: string (nullable = true)
> |-- quantity: decimal(20,4) (nullable = true)
> |-- unitprice: decimal(20,4) (nullable = true)
> |-- transno: string (nullable = true)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org