You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jkbradley <gi...@git.apache.org> on 2014/11/24 09:47:47 UTC
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
GitHub user jkbradley opened a pull request:
https://github.com/apache/spark/pull/3427
[MLLIB] [WIP] [SPARK-3702] Standardizing abstractions and developer API for prediction
This is WIP effort to standardize abstractions and developer API for prediction tasks (classification and regression) for the new ML api (org.apache.spark.ml).
* Please comment on:
* abstractions, class hierarchy
* functionality required by each abstraction
* naming of types and methods
* ease of use for developers
* ease of use for users migrating from org.apache.spark.mllib
* Please ignore for now:
* missing tests and examples
* private/public API (I will make more things private[ml] after writing tests and examples.)
* style and other details
* the many TODO items noted in the code
Please refer to [https://issues.apache.org/jira/browse/SPARK-3702] for some discussion on design, and [this design doc](https://docs.google.com/document/d/1I-8PD0DSLEZzzXURYZwmqAFn_OMBc08hgDL1FZnVBmw/) for major design decisions.
This is not intended to cover all algorithms; e.g., one big missing item is porting the GeneralizedLinearModel class to the new API. But it hopefully lays a fair amount of groundwork.
I have included a limited number of concrete classes in this WIP PR, for purposes of illustration:
* LogisticRegression (edited, to show effects of abstract classes)
* NaiveBayes (simple to show ease of use for developers)
* AdaBoost (demonstration of meta-algorithms taking advantage of abstractions)
* (Note discussion of strong vs. weak types for ensemble methods in design doc.)
* This implementation is very incomplete but illustrates using the abstractions.
* LinearRegression (example of Regressor, for completeness)
* evaluators (to provide default evaluators in the class hierarchy)
* IterativeSolver and IterativeEstimator (to expose iterative algorithms)
* LabeledPoint (Q: Should this include an instance weight?)
Items remaining:
- [ ] helper method for simulating a distribution over weighted instances by subsampling (for algorithms which do not support instance weights)
- [ ] several TODO items noted in the code
- [ ] add tests and examples
- [ ] general cleanup
- [ ] make more of hierarchy private[ml]
- [ ] split into several smaller PRs
Thanks to @epahomov and @BigCrunsh for input, including from [https://github.com/apache/spark/pull/2137] which improves upon the org.apache.spark.mllib APIs.
CC: @etrain @shivaram @mengxr
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkbradley/spark ml-api
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3427.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3427
----
commit f677562c91b7b90f13850fee089d9fc37d43d14c
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2014-11-24T07:18:52Z
Added lots of classes for new ML API:
Abstract classes for learning algorithms:
* Classifier
* Regressor
* Predictor
Traits for learning algorithms
* HasDefaultEstimator
* IterativeEstimator
* IterativeSolver
* ProbabilisticClassificationModel
* WeakLearner
Concrete classes: learning algorithms
* AdaBoost (partly implemented)
* NaiveBayes (rough implementation)
* LinearRegression
* LogisticRegression (updated to use new abstract classes)
Concrete classes: evaluation
* ClassificationEvaluator
* RegressionEvaluator
* PredictionEvaluator
Concrete classes: other
* LabeledPoint (adding weight to the old LabeledPoint)
commit 79f9fbca26a31a12e02f22442820097a42961499
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2014-11-24T07:30:31Z
fixed compilation issues, but have not added tests yet
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3427#issuecomment-67099848
I'm closing this since I've begun breaking it into smaller PRs. I copied the PR description to the JIRA and will leave my WIP branch intact.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley closed the pull request at:
https://github.com/apache/spark/pull/3427
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3427#issuecomment-64165818
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23780/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3427#issuecomment-64165814
[Test build #23780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23780/consoleFull) for PR 3427 at commit [`79f9fbc`](https://github.com/apache/spark/commit/79f9fbca26a31a12e02f22442820097a42961499).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class LabeledPoint(label: Double, features: Vector, weight: Double) `
* `class NaiveBayes extends Classifier[NaiveBayes, NaiveBayesModel] with NaiveBayesParams `
* `class ClassificationEvaluator extends PredictionEvaluator `
* `class RegressionEvaluator extends PredictionEvaluator `
* `trait HasDefaultEvaluator `
* `trait IterativeEstimator[M <: Model[M]] `
* `abstract class IterativeSolver[M <: Model[M]] `
* `trait WeakLearner[M <: PredictionModel[M]] `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3427#issuecomment-64165794
[Test build #23780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23780/consoleFull) for PR 3427 at commit [`79f9fbc`](https://github.com/apache/spark/commit/79f9fbca26a31a12e02f22442820097a42961499).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLLIB] [WIP] [SPARK-3702] Standardizing abstr...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/3427#issuecomment-66177125
I just submitted the first part of this PR: [https://github.com/apache/spark/pull/3637/files]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org