You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2016/07/11 00:35:11 UTC
[jira] [Commented] (SPARK-16467) After importing R data.frame,
although DataFrame columns show . replaced by _, the describe() function
gives warnings on . in the name
[ https://issues.apache.org/jira/browse/SPARK-16467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370028#comment-15370028 ]
Dongjoon Hyun commented on SPARK-16467:
---------------------------------------
Hi, you missed `"` in the last command. :)
{code}
> collect(describe(sdfIris, "Species"))
summary Species
1 count 150
2 mean <NA>
3 stddev <NA>
4 min setosa
5 max virginica
{code}
> After importing R data.frame, although DataFrame columns show . replaced by _, the describe() function gives warnings on . in the name
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-16467
> URL: https://issues.apache.org/jira/browse/SPARK-16467
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 1.6.1
> Reporter: Neil Dewar
> Priority: Minor
>
> When importing an R data.frame into a Spark DataFrame using createDataFrame(), if the R data.frame column names contain period characters "." they are converted to underscores "_". Most SparkR functions then display the column names with the underscores.
> If the describe() function is used, with the option for a specific function name, warning messages are displayed about the use of period characters.
> Example:
> sdfIris <- createDataFrame(sqlContext, iris)
> str(sdfIris) # works fine
> collect(describe(sdfIris, Species))
> the last function above throws a string of warnings such as:
> 1: In FUN(X[[i]], ...) : Use Sepal_Length instead of Sepal.Length as column name
> Note, the describe() function appears to work ok if no column name is specified but fails if a column name is specified, even if the specified name did not contain a period.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org