[jira] [Commented] (SPARK-16768) pyspark calls incorrect version of logistic regression

["Colin Beckingham (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-16768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402405#comment-15402405 ] 

Colin Beckingham commented on SPARK-16768:
------------------------------------------

OK, Sean, I've done some more reading and I note that ml is in the process of being preferred to MLLib. As a newcomer to Spark I am finding many resources using 1.6 version code with MLLib and naively thinking that the same code will work in 2.1; clearly it might and might not, have to be careful. My concern is that I have some code from a tutorial (spark-py-notebooks/nb8-mllib-logit/) which uses mllib which appears to work in both 1.6 and 2.1 without alteration, but the result produced in each case is different despite calling for L-BFGS in each version. Intuition tells me that the results should be the same. As and when I have more input to offer I will add it then.

> pyspark calls incorrect version of logistic regression
> ------------------------------------------------------
>
>                 Key: SPARK-16768
>                 URL: https://issues.apache.org/jira/browse/SPARK-16768
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib, PySpark
>         Environment: Linux openSUSE Leap 42.1 Gnome
>            Reporter: Colin Beckingham
>
> PySpark call with Spark 1.6.2 "LogisticRegressionWithLBFGS.train()"  runs "treeAggregate at LBFGS.scala:218" but the same command in pyspark with Spark 2.1 runs "treeAggregate at LogisticRegression.scala:1092". This non-optimized version is much slower and produces a different answer from LBFGS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org