You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by varadharajan <gi...@git.apache.org> on 2014/10/22 18:43:57 UTC
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
GitHub user varadharajan opened a pull request:
https://github.com/apache/spark/pull/2894
[SPARK-4047] - Generate runtime warnings for example implementation of PageRank
Based on SPARK-2434, this PR generates runtime warnings for example implementations (Python, Scala) of PageRank.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/varadharajan/spark SPARK-4047
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2894.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2894
----
commit 5c2bf5453834ad406a62d9e95d8ce128a65a2680
Author: Varadharajan Mukundan <sr...@gmail.com>
Date: 2014-10-22T16:47:59Z
[SPARK-4047] - Generate runtime warnings for example implementation of PageRank
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62262246
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23102/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by varadharajan <gi...@git.apache.org>.
Github user varadharajan commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-61589586
Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62468718
Merged into master and branch-1.2. Thanks! (We should find some time and clean really old examples.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-61032904
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22498/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-61027638
[Test build #22498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22498/consoleFull) for PR 2894 at commit [`252f595`](https://github.com/apache/spark/commit/252f595eece96771e385aab1ffce7031097b2c1a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by varadharajan <gi...@git.apache.org>.
Github user varadharajan commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62259256
@jkbradley Makes sense. I've updated the warnings, please let me know if wordings can be improved. Also i just noticed that pyspark classification model does not have LR-LBFGS implementation. I'll probably create a new issue and work on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by varadharajan <gi...@git.apache.org>.
Github user varadharajan commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62259405
Also i think it would help users if we can document in the LR section of the MLlib guide, which algorithm should be preferred in which scenarios.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62313396
@varadharajan Good suggestion about documenting algs for LR; I'll make a note to do that for the upcoming release. Thank you for the PR!
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by varadharajan <gi...@git.apache.org>.
Github user varadharajan commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62350096
@jkbradley Thanks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by varadharajan <gi...@git.apache.org>.
Github user varadharajan commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-60408059
Here are list of scala examples that i think is similar / naive implementation of algorithms from MLlib or graphx.
1. LocalALS
2. LocalFileLR
3. LocalKMeans
4. LocalLR
5. SparkALS
6. SparkHdfsLR
7. SparkKMeans
8. SparkLR
9. SparkPageRank (*)
10. SparkTachyonHdfsLR (*)
Python examples:
1. ALS
2. kmeans
3. logistic_regression
4. pagerank (*)
Java examples:
1. JavaHdfsLR (*)
2. JavaPageRank (*)
* - Examples with missing warnings. I've updated JIRA with these details and also added warning for them
I've also corrected class names of existing LR examples. They were pointing to org.apache.spark.mllib.classification.LogisticRegression instead of org.apache.spark.mllib.classification.LogisticRegressionModel
I've excluded examples that compute transitive closures on graphs because i'm was not able to find corresponding implementations in graphx. Please let me know if i'm missing something
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-61027290
Jenkins, ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-60394239
Are there other examples that should have the same warning? I think there are many more than this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-61032900
[Test build #22498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22498/consoleFull) for PR 2894 at commit [`252f595`](https://github.com/apache/spark/commit/252f595eece96771e385aab1ffce7031097b2c1a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62226699
@varadharajan Thanks for adding the warnings! My main comment is that LogisticRegressionModel is a model, rather than an algorithm. Users would really want the algorithm which they can run to produce the model. Could you instead direct users to the algorithms: LogisticRegressionWithSGD and LogisticRegressionWithLBFGS? (It is awkward that there are 2 algorithms to direct users towards, but it is hard to get around that.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62223607
Since this is MLlib related, @mengxr or @jkbradley, could one of you do the final sign-off + commit on this? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62259296
[Test build #23102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23102/consoleFull) for PR 2894 at commit [`5f9406b`](https://github.com/apache/spark/commit/5f9406b858ae1c80f228bd77d22bb1eebc6b1200).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-62262242
[Test build #23102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23102/consoleFull) for PR 2894 at commit [`5f9406b`](https://github.com/apache/spark/commit/5f9406b858ae1c80f228bd77d22bb1eebc6b1200).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2894
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-60116178
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4047] - Generate runtime warnings for e...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2894#issuecomment-61563547
LGTM, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org