[jira] [Commented] (SPARK-14659) OneHotEncoder support drop first category alphabetically in the encoded vector

["Yanbo Liang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-14659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242739#comment-15242739 ] 

Yanbo Liang commented on SPARK-14659:
-------------------------------------

For example:
{quote}
df=data.frame(id = c(1, 2, 3, 4), a = c("foo", "bar", "bar", "baz"), b = c(4, 4, 5, 5))
summary(glm(id ~ a + b, data = df))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)       -2         NA      NA       NA
abaz               1         NA      NA       NA
afoo              -1         NA      NA       NA
b                  1         NA      NA       NA
{quote}

R will drop "bar" when encode the string/category feature, because it's the first category alphabetically.
However, Spark RFormula will drop "baz" due to the lowest frequency. You can refer the RFormulaSuite(https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/feature/RFormulaSuite.scala#L92).

> OneHotEncoder support drop first category alphabetically in the encoded vector 
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-14659
>                 URL: https://issues.apache.org/jira/browse/SPARK-14659
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>            Reporter: Yanbo Liang
>
> R formula drop the first category alphabetically when encode string/category feature. Spark RFormula use OneHotEncoder to encode string/category feature into vector, but only supporting "dropLast" by string/category frequencies. This will cause SparkR produce different models compared with native R.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org