You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yanbo Liang (JIRA)" <ji...@apache.org> on 2016/04/15 12:09:25 UTC
[jira] [Comment Edited] (SPARK-14659) OneHotEncoder support drop
first category alphabetically in the encoded vector
[ https://issues.apache.org/jira/browse/SPARK-14659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242739#comment-15242739 ]
Yanbo Liang edited comment on SPARK-14659 at 4/15/16 10:08 AM:
---------------------------------------------------------------
Take the following R code as example:
{quote}
df=data.frame(id = c(1, 2, 3, 4), a = c("foo", "bar", "bar", "baz"), b = c(4, 4, 5, 5))
summary(glm(id ~ a + b, data = df))
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2 NA NA NA
abaz 1 NA NA NA
afoo -1 NA NA NA
b 1 NA NA NA
{quote}
R will drop "bar" when encode the string/category feature, because it's the first category alphabetically.
However, Spark RFormula will drop "baz" due to the lowest frequency. You can refer the RFormulaSuite(https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala#L92).
was (Author: yanboliang):
For example:
{quote}
df=data.frame(id = c(1, 2, 3, 4), a = c("foo", "bar", "bar", "baz"), b = c(4, 4, 5, 5))
summary(glm(id ~ a + b, data = df))
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2 NA NA NA
abaz 1 NA NA NA
afoo -1 NA NA NA
b 1 NA NA NA
{quote}
R will drop "bar" when encode the string/category feature, because it's the first category alphabetically.
However, Spark RFormula will drop "baz" due to the lowest frequency. You can refer the RFormulaSuite(https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala#L92).
> OneHotEncoder support drop first category alphabetically in the encoded vector
> -------------------------------------------------------------------------------
>
> Key: SPARK-14659
> URL: https://issues.apache.org/jira/browse/SPARK-14659
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Reporter: Yanbo Liang
>
> R formula drop the first category alphabetically when encode string/category feature. Spark RFormula use OneHotEncoder to encode string/category feature into vector, but only supporting "dropLast" by string/category frequencies. This will cause SparkR produce different models compared with native R.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org