You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hhbyyh <gi...@git.apache.org> on 2017/12/20 04:55:59 UTC
[GitHub] spark pull request #20028: [SPARK-19053][ML]Supporting multiple evaluation m...
GitHub user hhbyyh opened a pull request:
https://github.com/apache/spark/pull/20028
[SPARK-19053][ML]Supporting multiple evaluation metrics in DataFrame-based API
## What changes were proposed in this pull request?
As an initial step, the PR creates BinaryClassificationMetrics, MultiClassClassificationMetrics, RegressionMetrics in ML. The long term target is to reach function parity with MLlib Metrics and be able to provide more enhancements for DataFrame-based API.
This PR allows the Binary/MultilclassClassification/Regression Evaluator return a corresponding metrics instance, and users can use the metrics instance to access multiple metrics after a single pass, while originally Evaluator only allows access for single metric.
```
val evaluator = new MulticlassClassificationEvaluator().setMetricName("accuracy")
val metrics = evaluator.getMetrics(df)
metrics.accuracy
metrics.weightedFMeasure
metrics.weightedPrecision
metrics.weightedRecall
```
To make the review easier, the current PR only includes metrics in the Evaluator to reach function parity. Plan for further development:
1. Initial API and function parity with ML Evaluators. (This PR)
2. Python API.
2. Function parity with MLlib Metrics.
3. Add requested enhancements like including weight, add per-row metrics, add ranking metrics.
4. Reorganize classification Metrics hierarchy, so Binary Classification Metrics can support metrics in MultiClassMetrics (accuracy, recall etc.).
5. Possibly to be used in training summary.
The current implementation is still based on MLlib Metrics, which is kept completely internal and can be changed to DataFrame-based calculation when necessary.
## How was this patch tested?
new unit tests added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hhbyyh/spark mlMetrics
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20028.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20028
----
commit 5d5978e5c83bc2037646f40d6db23059af530a15
Author: Yuhao Yang <yu...@...>
Date: 2017-12-20T02:51:19Z
add ml metrics
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20028
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1509/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the issue:
https://github.com/apache/spark/pull/20028
Thanks for the comments @zhengruifeng @felixcheung
It's been nearly 8 months and it took me a while to recall what this PR does. While the PR did provide some improvement for the current API, I wonder if it lays a good foundation for an extensible and flexible `Evaluator` framework for Spark ML.
The current design is not quite user-friendly as it asks users to understand the concept of `Metrics` (BinaryClassificationMetrics, MultiClassClassificationMetrics, RegressionMetrics) which are primarily for internal calculation, and it implies that all the indicators in a `Metrics` can be calculated in one pass of the DataFrame, which creates some difficulty when we add extra indicators in the Metric which cannot be calculated with other indicators.
IMO, API wise, ideally we should allow users to specify any combination of the metrics that they want to add to the `Evaluator`, then the `Evaluator` needs to figure out the best way to efficiently calculate the metrics. Following are the concrete suggestions:
1. Evaluator API:
```
ClassificationEvaluator {
def setPredictionCol(value: String): this.type
def setLabelCol(value: String): this.type
// kept for back-ward compatibility and Cross validation
def setMetricName(value: String): this.type
// kept for back-ward compatibility and Cross validation
override def evaluate(dataset: Dataset[_]): Double
// calculate multiple metrics, will try to optimize calculation internally
override def getMetrics(dataset: Dataset[_], metrics: Array[String]): Map[String, Any] // or wrap it with customized class
}
val ce = new ClassificationEvaluator().setLabelCol("x").setPredictionCol("y")
metrics = ce.getMetrics(dataframe, Array(Classification.truePositiveRateByLabel, BinaryClassification.areaUnderROC))
println(metrics)
```
We can basically merge BinaryClassificationEvaluator and MultiClassificationEvaluator.
Similarly we can have `RegressionEvaluator` and `ClusteringEvaluator`, separating those because we may need to provide different setters in each.
2. Summy classes may invoke Evaluator internally.
@felixcheung , I'm not sure if this can get a shepherd and review bandwidth for the next release. I don't want to just update version numbers every a few months.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:
https://github.com/apache/spark/pull/20028
LGTM, except for the since annotations.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20028
**[Test build #85155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85155/testReport)** for PR 20028 at commit [`5d5978e`](https://github.com/apache/spark/commit/5d5978e5c83bc2037646f40d6db23059af530a15).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20028
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85155/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the issue:
https://github.com/apache/spark/pull/20028
Please advice if this is a good feature to add. If not I'll close it. Thanks.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20028
this shouldn't say version 2.3.0?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20028
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20028
**[Test build #85155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85155/testReport)** for PR 20028 at commit [`5d5978e`](https://github.com/apache/spark/commit/5d5978e5c83bc2037646f40d6db23059af530a15).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20028: [SPARK-19053][ML]Supporting multiple evaluation metrics ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20028
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org