You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hhbyyh <gi...@git.apache.org> on 2015/11/11 03:20:41 UTC
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
GitHub user hhbyyh opened a pull request:
https://github.com/apache/spark/pull/9614
[SPARK-11579] [ML] avoid creating new optimizer in ANN
jira: https://issues.apache.org/jira/browse/SPARK-11579
This is a proposal based on some customer feedback.
Currently method SGDOptimizer and LBFGSOptimizer in FeedForwardTrainer create new optimizer every time they got invoked, this is not quite intuitive since users think they are still using the existing optimizer when they write:
```
feedForwardTrainer
.SGDOptimizer
.setMiniBatchFraction(0.002)
```
yet it actually creates a new optimizer without other properties which were set previously.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hhbyyh/spark annOptimizer
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9614.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9614
----
commit 2c9745d09417201db5dc12f6aa721d671e53d8f3
Author: Yuhao Yang <hh...@gmail.com>
Date: 2015-11-11T01:51:33Z
avoid unnecessary new optimizer
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-155633921
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh closed the pull request at:
https://github.com/apache/spark/pull/9614
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-155633985
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-155638397
**[Test build #45582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45582/consoleFull)** for PR 9614 at commit [`2c9745d`](https://github.com/apache/spark/commit/2c9745d09417201db5dc12f6aa721d671e53d8f3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-155648868
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45582/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-169191098
OK. I'll close this. Thanks for helping review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by avulanov <gi...@git.apache.org>.
Github user avulanov commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168854243
@jkbradley If we want to follow the logic "users think they are still using the existing optimizer when they access the optimizer", then there should be two private variables for each optimizer instead of one. The proposed fix creates a new optimizer if user switches from SGD to LBFGS and the mentioned logic breaks.
By the way, `FeedForwardTrainer` is `ML` private, so it should not be used by users. In practice, the creation of new optimizer never happens because the public class `MultilayerPerceptronClassifier` uses LBFGS only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168851599
@avulanov Could you please take a look? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-169110930
@avulanov Thanks that makes sense.
@hhbyyh I'd agree with not creating a new optimizer every time, if this were a public API. Is there a real need for it to be public currently?
Eventually, it'd be nice to have it public and have it follow the same logic as GLM optimizers. But perhaps we can skip this for now?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168935901
@avulanov Thanks for review.
The only possible concern I got is LBFGSOptimizer/SGDOptimizer appears to be getter, yet they are setters.
I sent an update which changes the function name only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168946249
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168945836
**[Test build #48753 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48753/consoleFull)** for PR 9614 at commit [`dcf0d8f`](https://github.com/apache/spark/commit/dcf0d8ff111cdb6812cb5ff74d0119331270b644).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168946255
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48753/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-155648782
**[Test build #45582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45582/consoleFull)** for PR 9614 at commit [`2c9745d`](https://github.com/apache/spark/commit/2c9745d09417201db5dc12f6aa721d671e53d8f3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-168934988
**[Test build #48753 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48753/consoleFull)** for PR 9614 at commit [`dcf0d8f`](https://github.com/apache/spark/commit/dcf0d8ff111cdb6812cb5ff74d0119331270b644).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-11579] [ML] avoid creating new optimize...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9614#issuecomment-155648865
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org