You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by philippe v <gl...@gmail.com> on 2016/06/08 08:22:05 UTC
Trainning a spark ml linear regresion model fail after migrating
from 1.5.2 to 1.6.1
I use spark-ml to train a linear regression model. It worked perfectly with
spark version 1.5.2 but now with 1.6.1 I get the following error :
Here is a minimal code :
And input.csv data
the pom.xml
How can I fix it ?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Trainning a spark ml linear regresion model fail after migrating
from 1.5.2 to 1.6.1
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi,
Is it me only to *not* see the snippets? Could you please gist 'em =>
https://gist.github.com ?
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
On Wed, Jun 8, 2016 at 10:22 AM, philippe v <gl...@gmail.com> wrote:
> I use spark-ml to train a linear regression model. It worked perfectly with
> spark version 1.5.2 but now with 1.6.1 I get the following error :
>
>
>
> Here is a minimal code :
>
>
>
> And input.csv data
>
>
>
> the pom.xml
>
>
>
>
> How can I fix it ?
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Trainning a spark ml linear regresion model fail after migrating
from 1.5.2 to 1.6.1
Posted by Yanbo Liang <yb...@gmail.com>.
Yes, WeightedLeastSquares can not solve some ill-conditioned problem
currently, the community members have paid some efforts to resolve it
(SPARK-13777). For the work around, you can set the solver to "l-bfgs"
which will train the LogisticRegressionModel by L-BFGS optimization method.
2016-06-09 7:37 GMT-07:00 chaz2505 <ch...@hotmail.com>:
> I ran into this problem too - it's because WeightedLeastSquares (added in
> 1.6.0 SPARK-10668) is being used on an ill-conditioned problem
> (SPARK-11918). I guess because of the one hot encoding. To get around it
> you
> need to ensure WeightedLeastSquares isn't used. Set parameters to make the
> following false:
>
> $(solver) == "auto" && $(elasticNetParam) == 0.0 &&
> numFeatures <= WeightedLeastSquares.MAX_NUM_FEATURES) || $(solver) ==
> "normal"
>
> Hope this helps
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111p27128.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Trainning a spark ml linear regresion model fail after
migrating from 1.5.2 to 1.6.1
Posted by chaz2505 <ch...@hotmail.com>.
I ran into this problem too - it's because WeightedLeastSquares (added in
1.6.0 SPARK-10668) is being used on an ill-conditioned problem
(SPARK-11918). I guess because of the one hot encoding. To get around it you
need to ensure WeightedLeastSquares isn't used. Set parameters to make the
following false:
$(solver) == "auto" && $(elasticNetParam) == 0.0 &&
numFeatures <= WeightedLeastSquares.MAX_NUM_FEATURES) || $(solver) ==
"normal"
Hope this helps
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111p27128.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Trainning a spark ml linear regresion model fail after
migrating from 1.5.2 to 1.6.1
Posted by philippe v <gl...@gmail.com>.
here is a gist with the minimal code and data
http://gist.github.com/anonymous/aca8ba5841404ea092f9efcc658c5d57
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trainning-a-spark-ml-linear-regresion-model-fail-after-migrating-from-1-5-2-to-1-6-1-tp27111p27116.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org