You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yanbo Liang (JIRA)" <ji...@apache.org> on 2016/04/15 11:40:25 UTC
[jira] [Comment Edited] (SPARK-14657) RFormula output wrong
features when formula w/o intercept
[ https://issues.apache.org/jira/browse/SPARK-14657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242701#comment-15242701 ]
Yanbo Liang edited comment on SPARK-14657 at 4/15/16 9:39 AM:
--------------------------------------------------------------
More cases in R:
df=data.frame(income=c(5,5,3,3,6,5),
won=c(0,0,1,1,1,0),
age=c(18,18,23,50,19,39),
home=c(0,0,1,0,0,1))
{quote}
model <- glm(factor(won) ~ factor(income) + factor(age) + factor(home), data=df, family=binomial(link="logit"))
Coefficients: (3 not defined because of singularities)
Estimate Std. Error z value Pr(>|z|)
(Intercept) 2.457e+01 1.310e+05 0 1
factor(income)5 -4.913e+01 1.605e+05 0 1
factor(income)6 -2.573e-30 1.853e+05 0 1
factor(age)19 NA NA NA NA
factor(age)23 -1.383e-30 1.853e+05 0 1
factor(age)39 -3.479e-14 1.605e+05 0 1
factor(age)50 NA NA NA NA
factor(home)1 NA NA NA NA
{quote}
model2 <- glm(factor(won) ~ factor(income) + factor(age) + factor(home) - 1, data=df, family=binomial(link="logit"))
model3 <- glm(factor(won) ~ factor(age) + factor(income) + factor(home) - 1, data=df, family=binomial(link="logit"))
was (Author: yanboliang):
More cases in R:
{quote}
df=data.frame(income=c(5,5,3,3,6,5),
won=c(0,0,1,1,1,0),
age=c(18,18,23,50,19,39),
home=c(0,0,1,0,0,1))
model <- glm(factor(won) ~ factor(income) + factor(age) + factor(home), data=df, family=binomial(link="logit"))
{quote}
Coefficients: (3 not defined because of singularities)
Estimate Std. Error z value Pr(>|z|)
(Intercept) 2.457e+01 1.310e+05 0 1
factor(income)5 -4.913e+01 1.605e+05 0 1
factor(income)6 -2.573e-30 1.853e+05 0 1
factor(age)19 NA NA NA NA
factor(age)23 -1.383e-30 1.853e+05 0 1
factor(age)39 -3.479e-14 1.605e+05 0 1
factor(age)50 NA NA NA NA
factor(home)1 NA NA NA NA
{quote}
model2 <- glm(factor(won) ~ factor(income) + factor(age) + factor(home) - 1, data=df, family=binomial(link="logit"))
model3 <- glm(factor(won) ~ factor(age) + factor(income) + factor(home) - 1, data=df, family=binomial(link="logit"))
> RFormula output wrong features when formula w/o intercept
> ---------------------------------------------------------
>
> Key: SPARK-14657
> URL: https://issues.apache.org/jira/browse/SPARK-14657
> Project: Spark
> Issue Type: Bug
> Components: ML
> Reporter: Yanbo Liang
>
> SparkR::glm output different features compared with R glm.
> SparkR::glm
> {quote}
> training <- suppressWarnings(createDataFrame(sqlContext, iris))
> model <- glm(Sepal_Width ~ Sepal_Length + Species - 1, data = training)
> summary(model)
> Coefficients:
> Estimate Std. Error t value Pr(>|t|)
> Sepal_Length 0.67468 0.0093013 72.536 0
> Species_versicolor -1.2349 0.07269 -16.989 0
> Species_virginica -1.4708 0.077397 -19.003 0
> {quote}
> stats::glm
> {quote}
> summary(glm(Sepal.Width ~ Sepal.Length + Species - 1, data = iris))
> Coefficients:
> Estimate Std. Error t value Pr(>|t|)
> Sepal.Length 0.3499 0.0463 7.557 4.19e-12 ***
> Speciessetosa 1.6765 0.2354 7.123 4.46e-11 ***
> Speciesversicolor 0.6931 0.2779 2.494 0.0137 *
> Speciesvirginica 0.6690 0.3078 2.174 0.0313 *
> {quote}
> The encoder for string type feature is different. R did not drop any category but SparkR drop the last one.
> I searched online and test some other cases, found when we fit R glm model(or other models powered by R formula) w/o intercept on a dataset including string/category features, one of the levels in the first category feature is being used as reference level, we will not drop any category for that feature.
> I think we should keep consistent semantics between Spark RFormula and R formula.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org