You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nsareen <ns...@gmail.com> on 2016/09/02 01:50:45 UTC
MLib : Non Linear Optimization
I'm part of an Predictive Analytics marketing platform. We do a lot of
Optimizations ( non linear ), currently using SAS / Lindo routines. I was
going through Spark's MLib documentation & found it supports Linear
Optimization, was wondering if it also supports Non Linear Optimization & if
not, are there any plans to implement it in spark ? We really want to move
away from SAS since it is a very expensive solution & does not work on a
distributed scale. We want a solution which provides scalability & at the
same time provide accurate results.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: MLib : Non Linear Optimization
Posted by Nitin Sareen <ns...@gmail.com>.
Yes, we are using primarily these two algorithms.
1. Interior point trust-region line-search algorithm
2. Active-set trust-region line-search algorithm
We are performing optimizations with constraints & thresholds etc.
We are primarily using Lindo / SAS modules but want to get away from SAS
due to the cost, it would be really good to have these algorithms in Spark
ML.
Let me know if you need any more info, i can share some snippets if
required.
Thanks,
Nitin
On Thu, Sep 8, 2016 at 2:08 PM, Robin East <ro...@xense.co.uk> wrote:
> Do you have any particular algorithms in mind? If you state the most
> common algorithms you use then it might stimulate the appropriate comments.
>
>
>
> > On 8 Sep 2016, at 05:04, nsareen <ns...@gmail.com> wrote:
> >
> > Any answer to this question group ?
> >
> >
> >
> > --
> > View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27676.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
>
>
Re: MLib : Non Linear Optimization
Posted by Robin East <ro...@xense.co.uk>.
Do you have any particular algorithms in mind? If you state the most common algorithms you use then it might stimulate the appropriate comments.
> On 8 Sep 2016, at 05:04, nsareen <ns...@gmail.com> wrote:
>
> Any answer to this question group ?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27676.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: MLib : Non Linear Optimization
Posted by Blake Ryan <bl...@gmail.com>.
Outside of Spark, we'd recommend looking at IpOpt as an open source
alternative for this: http://www.coin-or.org/projects/Ipopt.xml
Blake
On Wed, Oct 5, 2016 at 8:59 AM Robin East <ro...@xense.co.uk> wrote:
> The TFOCS package is announced here:
> https://databricks.com/blog/2015/11/02/announcing-the-spark-tfocs-optimization-package.html
>
>
> -------------------------------------------------------------------------------
> Robin East
> *Spark GraphX in Action* Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/books/spark-graphx-in-action
>
>
>
>
>
> On 5 Oct 2016, at 08:29, Robin East <ro...@xense.co.uk> wrote:
>
> I would say no, at least not without a fair degree of algorithm writing
> experience. MLLib is primarily a set of machine learning algorithms, many
> of which are based on implementations of distributed optimisation
> procedures. The SAS routines you mention are optimisation routines which
> don't have directly comparable implementations in MLLib. One possibility is
> the TFOCS spark package which might be of interest but I'm not sure it
> quite matches what you are asking for. It's built by databricks so maybe
> one of their people can advise.
>
> One question I would ask is is this a big data problem? Could you use a
> python or julia library on a machine with large amounts of RAM?
>
> Would be interested in hearing the views of others on the forum.
>
> Sent from my iPhone
>
> On 5 Oct 2016, at 05:28, nsareen <ns...@gmail.com> wrote:
>
> I'm not getting any support in this group, is the question not valid ? need
> someone to reply to this question, we have a huge dependency on SAS which
> we
> want to eliminate & want to know if spark can help.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27835.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com
> <http://nabble.com>.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>
Re: MLib : Non Linear Optimization
Posted by Robin East <ro...@xense.co.uk>.
The TFOCS package is announced here: https://databricks.com/blog/2015/11/02/announcing-the-spark-tfocs-optimization-package.html <https://databricks.com/blog/2015/11/02/announcing-the-spark-tfocs-optimization-package.html>
-------------------------------------------------------------------------------
Robin East
Spark GraphX in Action Michael Malak and Robin East
Manning Publications Co.
http://www.manning.com/books/spark-graphx-in-action <http://www.manning.com/books/spark-graphx-in-action>
> On 5 Oct 2016, at 08:29, Robin East <ro...@xense.co.uk> wrote:
>
> I would say no, at least not without a fair degree of algorithm writing experience. MLLib is primarily a set of machine learning algorithms, many of which are based on implementations of distributed optimisation procedures. The SAS routines you mention are optimisation routines which don't have directly comparable implementations in MLLib. One possibility is the TFOCS spark package which might be of interest but I'm not sure it quite matches what you are asking for. It's built by databricks so maybe one of their people can advise.
>
> One question I would ask is is this a big data problem? Could you use a python or julia library on a machine with large amounts of RAM?
>
> Would be interested in hearing the views of others on the forum.
>
> Sent from my iPhone
>
>> On 5 Oct 2016, at 05:28, nsareen <ns...@gmail.com> wrote:
>>
>> I'm not getting any support in this group, is the question not valid ? need
>> someone to reply to this question, we have a huge dependency on SAS which we
>> want to eliminate & want to know if spark can help.
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27835.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
Re: MLib : Non Linear Optimization
Posted by Robin East <ro...@xense.co.uk>.
I would say no, at least not without a fair degree of algorithm writing experience. MLLib is primarily a set of machine learning algorithms, many of which are based on implementations of distributed optimisation procedures. The SAS routines you mention are optimisation routines which don't have directly comparable implementations in MLLib. One possibility is the TFOCS spark package which might be of interest but I'm not sure it quite matches what you are asking for. It's built by databricks so maybe one of their people can advise.
One question I would ask is is this a big data problem? Could you use a python or julia library on a machine with large amounts of RAM?
Would be interested in hearing the views of others on the forum.
Sent from my iPhone
> On 5 Oct 2016, at 05:28, nsareen <ns...@gmail.com> wrote:
>
> I'm not getting any support in this group, is the question not valid ? need
> someone to reply to this question, we have a huge dependency on SAS which we
> want to eliminate & want to know if spark can help.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27835.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: MLib : Non Linear Optimization
Posted by nsareen <ns...@gmail.com>.
I'm not getting any support in this group, is the question not valid ? need
someone to reply to this question, we have a huge dependency on SAS which we
want to eliminate & want to know if spark can help.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27835.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: MLib : Non Linear Optimization
Posted by nsareen <ns...@gmail.com>.
Any answer to this question group ?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLib-Non-Linear-Optimization-tp27645p27676.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org