You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "k.tham" <ke...@gmail.com> on 2014/10/24 23:09:40 UTC
Re: Spark LIBLINEAR
Just wondering, any update on this? Is there a plan to integrate CJ's work
with mllib? I'm asking since SVM impl in MLLib did not give us good results
and we have to resort to training our svm classifier in a serial manner on
the driver node with liblinear.
Also, it looks like CJ Lin is coming to the bay area in the coming weeks
(http://www.meetup.com/sfmachinelearning/events/208078582/) might be a good
time to connect with him.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17236.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark LIBLINEAR
Posted by Debasish Das <de...@gmail.com>.
Hi Professor Lin,
It will be great if you could please review the TRON code in breeze and
whether it is similar to the original TRON implementation...Breeze is
already integrated in mllib (we are using BFGS and OWLQN is under works for
mllib LogisticRegression) and comparing with TRON should be quick...
The code is over here:
https://github.com/scalanlp/breeze/blob/master/math/src/main/scala/breeze/optimize/TruncatedNewtonMinimizer.scala
If there are bugs, fixing them in breeze will be very helpful for people
who are building upon it (like me :-)
Thanks.
Deb
On Sun, Oct 26, 2014 at 6:33 PM, Chih-Jen Lin <cj...@csie.ntu.edu.tw> wrote:
> Debasish Das writes:
> > If the SVM is not already migrated to BFGS, that's the first thing you
> should
> > try...Basically following LBFGS Logistic Regression come up with LBFGS
> based
> > linear SVM...
> >
> > About integrating TRON in mllib, David already has a version of TRON in
> breeze
> > but someone needs to validate it for linear SVM and do experiment to
> see if it
> > can improve upon LBFGS based linear SVM...Based on lib-linear papers,
> it should
> > but I don't expect substantial difference...
> >
> > I am validating TRON for use-cases related to this PR (but I need more
> features
> > on top of TRON):
> >
> > https://github.com/apache/spark/pull/2705
> >
>
> We are also working on integrating TRON to MLlib, though we haven't cheked
> the above work.
>
> We are also doing some serious comparison between quasi Newton and Newton,
> though this will take some time
>
> Chih-Jen
> > On Fri, Oct 24, 2014 at 2:09 PM, k.tham <ke...@gmail.com> wrote:
> >
> > Just wondering, any update on this? Is there a plan to integrate
> CJ's work
> > with mllib? I'm asking since SVM impl in MLLib did not give us good
> > results
> > and we have to resort to training our svm classifier in a serial
> manner on
> > the driver node with liblinear.
> >
> > Also, it looks like CJ Lin is coming to the bay area in the coming
> weeks
> > (http://www.meetup.com/sfmachinelearning/events/208078582/) might
> be a good
> > time to connect with him.
> >
> > --
> > View this message in context: http://
> > apache-spark-user-list.1001560.n3.nabble.com/
> > Spark-LIBLINEAR-tp5546p17236.html
> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
Re: Spark LIBLINEAR
Posted by Chih-Jen Lin <cj...@csie.ntu.edu.tw>.
Debasish Das writes:
> If the SVM is not already migrated to BFGS, that's the first thing you should
> try...Basically following LBFGS Logistic Regression come up with LBFGS based
> linear SVM...
>
> About integrating TRON in mllib, David already has a version of TRON in breeze
> but someone needs to validate it for linear SVM and do experiment to see if it
> can improve upon LBFGS based linear SVM...Based on lib-linear papers, it should
> but I don't expect substantial difference...
>
> I am validating TRON for use-cases related to this PR (but I need more features
> on top of TRON):
>
> https://github.com/apache/spark/pull/2705
>
We are also working on integrating TRON to MLlib, though we haven't cheked
the above work.
We are also doing some serious comparison between quasi Newton and Newton,
though this will take some time
Chih-Jen
> On Fri, Oct 24, 2014 at 2:09 PM, k.tham <ke...@gmail.com> wrote:
>
> Just wondering, any update on this? Is there a plan to integrate CJ's work
> with mllib? I'm asking since SVM impl in MLLib did not give us good
> results
> and we have to resort to training our svm classifier in a serial manner on
> the driver node with liblinear.
>
> Also, it looks like CJ Lin is coming to the bay area in the coming weeks
> (http://www.meetup.com/sfmachinelearning/events/208078582/) might be a good
> time to connect with him.
>
> --
> View this message in context: http://
> apache-spark-user-list.1001560.n3.nabble.com/
> Spark-LIBLINEAR-tp5546p17236.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark LIBLINEAR
Posted by DB Tsai <db...@dbtsai.com>.
yeah, column normalizarion. for some of the datasets, without doing
this, it will not be converged.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Fri, Oct 24, 2014 at 3:46 PM, Debasish Das <de...@gmail.com> wrote:
> You mean row/column normalization of data ? how much performance gain you
> saw using that ?
>
>
> On Fri, Oct 24, 2014 at 3:14 PM, DB Tsai <db...@dbtsai.com> wrote:
>>
>> oh, we just train the model in the standardized space which will help
>> the convergence of LBFGS. Then we convert the weights to original
>> space so the whole thing is transparent to users.
>>
>> Sincerely,
>>
>> DB Tsai
>> -------------------------------------------------------
>> My Blog: https://www.dbtsai.com
>> LinkedIn: https://www.linkedin.com/in/dbtsai
>>
>>
>> On Fri, Oct 24, 2014 at 3:13 PM, Debasish Das <de...@gmail.com>
>> wrote:
>> > @dbtsai for condition number what did you use ? Diagonal preconditioning
>> > of
>> > the inverse of B matrix ? But then B matrix keeps on changing...did u
>> > condition it after every few iterations ?
>> >
>> > Will it be possible to put that code in Breeze since it will be very
>> > useful
>> > to condition other solvers as well...
>> >
>> > On Fri, Oct 24, 2014 at 3:02 PM, DB Tsai <db...@dbtsai.com> wrote:
>> >>
>> >> We don't have SVMWithLBFGS, but you can check out how we implement
>> >> LogisticRegressionWithLBFGS, and we also deal with some condition
>> >> number improving stuff in LogisticRegressionWithLBFGS which improves
>> >> the performance dramatically.
>> >>
>> >> Sincerely,
>> >>
>> >> DB Tsai
>> >> -------------------------------------------------------
>> >> My Blog: https://www.dbtsai.com
>> >> LinkedIn: https://www.linkedin.com/in/dbtsai
>> >>
>> >>
>> >> On Fri, Oct 24, 2014 at 2:39 PM, k.tham <ke...@gmail.com> wrote:
>> >> > Oh, I've only seen SVMWithSGD, hadn't realized LBFGS was implemented.
>> >> > I'll
>> >> > try it out when I have time. Thanks!
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > View this message in context:
>> >> >
>> >> > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17240.html
>> >> > Sent from the Apache Spark User List mailing list archive at
>> >> > Nabble.com.
>> >> >
>> >> > ---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> >> > For additional commands, e-mail: user-help@spark.apache.org
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: user-help@spark.apache.org
>> >>
>> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark LIBLINEAR
Posted by Debasish Das <de...@gmail.com>.
You mean row/column normalization of data ? how much performance gain you
saw using that ?
On Fri, Oct 24, 2014 at 3:14 PM, DB Tsai <db...@dbtsai.com> wrote:
> oh, we just train the model in the standardized space which will help
> the convergence of LBFGS. Then we convert the weights to original
> space so the whole thing is transparent to users.
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Fri, Oct 24, 2014 at 3:13 PM, Debasish Das <de...@gmail.com>
> wrote:
> > @dbtsai for condition number what did you use ? Diagonal preconditioning
> of
> > the inverse of B matrix ? But then B matrix keeps on changing...did u
> > condition it after every few iterations ?
> >
> > Will it be possible to put that code in Breeze since it will be very
> useful
> > to condition other solvers as well...
> >
> > On Fri, Oct 24, 2014 at 3:02 PM, DB Tsai <db...@dbtsai.com> wrote:
> >>
> >> We don't have SVMWithLBFGS, but you can check out how we implement
> >> LogisticRegressionWithLBFGS, and we also deal with some condition
> >> number improving stuff in LogisticRegressionWithLBFGS which improves
> >> the performance dramatically.
> >>
> >> Sincerely,
> >>
> >> DB Tsai
> >> -------------------------------------------------------
> >> My Blog: https://www.dbtsai.com
> >> LinkedIn: https://www.linkedin.com/in/dbtsai
> >>
> >>
> >> On Fri, Oct 24, 2014 at 2:39 PM, k.tham <ke...@gmail.com> wrote:
> >> > Oh, I've only seen SVMWithSGD, hadn't realized LBFGS was implemented.
> >> > I'll
> >> > try it out when I have time. Thanks!
> >> >
> >> >
> >> >
> >> > --
> >> > View this message in context:
> >> >
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17240.html
> >> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> > For additional commands, e-mail: user-help@spark.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: user-help@spark.apache.org
> >>
> >
>
Re: Spark LIBLINEAR
Posted by DB Tsai <db...@dbtsai.com>.
oh, we just train the model in the standardized space which will help
the convergence of LBFGS. Then we convert the weights to original
space so the whole thing is transparent to users.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Fri, Oct 24, 2014 at 3:13 PM, Debasish Das <de...@gmail.com> wrote:
> @dbtsai for condition number what did you use ? Diagonal preconditioning of
> the inverse of B matrix ? But then B matrix keeps on changing...did u
> condition it after every few iterations ?
>
> Will it be possible to put that code in Breeze since it will be very useful
> to condition other solvers as well...
>
> On Fri, Oct 24, 2014 at 3:02 PM, DB Tsai <db...@dbtsai.com> wrote:
>>
>> We don't have SVMWithLBFGS, but you can check out how we implement
>> LogisticRegressionWithLBFGS, and we also deal with some condition
>> number improving stuff in LogisticRegressionWithLBFGS which improves
>> the performance dramatically.
>>
>> Sincerely,
>>
>> DB Tsai
>> -------------------------------------------------------
>> My Blog: https://www.dbtsai.com
>> LinkedIn: https://www.linkedin.com/in/dbtsai
>>
>>
>> On Fri, Oct 24, 2014 at 2:39 PM, k.tham <ke...@gmail.com> wrote:
>> > Oh, I've only seen SVMWithSGD, hadn't realized LBFGS was implemented.
>> > I'll
>> > try it out when I have time. Thanks!
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17240.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark LIBLINEAR
Posted by Debasish Das <de...@gmail.com>.
@dbtsai for condition number what did you use ? Diagonal preconditioning of
the inverse of B matrix ? But then B matrix keeps on changing...did u
condition it after every few iterations ?
Will it be possible to put that code in Breeze since it will be very useful
to condition other solvers as well...
On Fri, Oct 24, 2014 at 3:02 PM, DB Tsai <db...@dbtsai.com> wrote:
> We don't have SVMWithLBFGS, but you can check out how we implement
> LogisticRegressionWithLBFGS, and we also deal with some condition
> number improving stuff in LogisticRegressionWithLBFGS which improves
> the performance dramatically.
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Fri, Oct 24, 2014 at 2:39 PM, k.tham <ke...@gmail.com> wrote:
> > Oh, I've only seen SVMWithSGD, hadn't realized LBFGS was implemented.
> I'll
> > try it out when I have time. Thanks!
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17240.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Spark LIBLINEAR
Posted by DB Tsai <db...@dbtsai.com>.
We don't have SVMWithLBFGS, but you can check out how we implement
LogisticRegressionWithLBFGS, and we also deal with some condition
number improving stuff in LogisticRegressionWithLBFGS which improves
the performance dramatically.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Fri, Oct 24, 2014 at 2:39 PM, k.tham <ke...@gmail.com> wrote:
> Oh, I've only seen SVMWithSGD, hadn't realized LBFGS was implemented. I'll
> try it out when I have time. Thanks!
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17240.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark LIBLINEAR
Posted by "k.tham" <ke...@gmail.com>.
Oh, I've only seen SVMWithSGD, hadn't realized LBFGS was implemented. I'll
try it out when I have time. Thanks!
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17240.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark LIBLINEAR
Posted by Debasish Das <de...@gmail.com>.
If the SVM is not already migrated to BFGS, that's the first thing you
should try...Basically following LBFGS Logistic Regression come up with
LBFGS based linear SVM...
About integrating TRON in mllib, David already has a version of TRON in
breeze but someone needs to validate it for linear SVM and do experiment to
see if it can improve upon LBFGS based linear SVM...Based on lib-linear
papers, it should but I don't expect substantial difference...
I am validating TRON for use-cases related to this PR (but I need more
features on top of TRON):
https://github.com/apache/spark/pull/2705
On Fri, Oct 24, 2014 at 2:09 PM, k.tham <ke...@gmail.com> wrote:
> Just wondering, any update on this? Is there a plan to integrate CJ's work
> with mllib? I'm asking since SVM impl in MLLib did not give us good
> results
> and we have to resort to training our svm classifier in a serial manner on
> the driver node with liblinear.
>
> Also, it looks like CJ Lin is coming to the bay area in the coming weeks
> (http://www.meetup.com/sfmachinelearning/events/208078582/) might be a
> good
> time to connect with him.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-LIBLINEAR-tp5546p17236.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>