You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Stephen Boesch <ja...@gmail.com> on 2016/06/23 23:56:56 UTC

Custom Optimizer

My team has a custom optimization routine that we would have wanted to plug
in as a replacement for the default  LBFGS /  OWLQN for use by some of the
ml/mllib algorithms.

However it seems the choice of optimizer is hard-coded in every algorithm
except LDA: and even in that one it is only a choice between the internally
defined Online or batch version.

Any suggestions on how we might be able to incorporate our own optimizer?
Or do we need to roll all of our algorithms from top to bottom - basically
side stepping ml/mllib?

thanks
stephen

Re: Custom Optimizer

Posted by Yanbo Liang <yb...@gmail.com>.

Spark MLlib does not support optimizer as a plugin, since the optimizer
interface is private.

Thanks
Yanbo

2016-06-23 16:56 GMT-07:00 Stephen Boesch <ja...@gmail.com>:

> My team has a custom optimization routine that we would have wanted to
> plug in as a replacement for the default  LBFGS /  OWLQN for use by some of
> the ml/mllib algorithms.
>
> However it seems the choice of optimizer is hard-coded in every algorithm
> except LDA: and even in that one it is only a choice between the internally
> defined Online or batch version.
>
> Any suggestions on how we might be able to incorporate our own optimizer?
> Or do we need to roll all of our algorithms from top to bottom - basically
> side stepping ml/mllib?
>
> thanks
> stephen
>