You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Neil Dewar (JIRA)" <ji...@apache.org> on 2016/07/15 03:02:20 UTC
[jira] [Closed] (SPARK-16468) Confusing results when describe()
used on DataFrame with chr columns
[ https://issues.apache.org/jira/browse/SPARK-16468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neil Dewar closed SPARK-16468.
------------------------------
Resolution: Not A Problem
Agreed that the problem can be addressed by formatting the response by display()
> Confusing results when describe() used on DataFrame with chr columns
> --------------------------------------------------------------------
>
> Key: SPARK-16468
> URL: https://issues.apache.org/jira/browse/SPARK-16468
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 1.6.1
> Environment: Databricks.com
> Reporter: Neil Dewar
> Priority: Minor
>
> The describe() function returns statistical summaries on numeric columns of a DataFrame. If the DataFrame contains columns of type chr, only the count, min and max stats are returned.
> When a dataframe contains a mixture of numeric and chr columns, the results become jumbled together.
> Example:
> sdfR <- createDataFrame(sqlContext, ToothGrowth)
> collect(describe(sdfR))
> Results:
> summary len supp dose
> 1 count 60 60 60
> 2 mean 18.813333333333336 1.1666666666666667
> 3 stddev 7.649315171887615 0.6288721857330792
> 4 min 4.2 OJ 0.5
> 5 max 33.9 VC 2.0
> There appear to be two problems here:
> (1) The mean and stdev values have not been rounded for the columns where there are valid values
> (2) There is no ability to distinguish that the supp column has no values in mean and stdev rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org