Re: mllib metrics vs ml evaluators and how to improve apis for users

[Joseph Bradley <jo...@databricks.com>]

Hi Ilya,

Thanks for your thoughts.  Here's my understanding of where we are headed:
* We will want to move the *Metrics functionality to the spark.ml package,
as part of *Evaluator or related classes such as model/result summaries.
* It has not yet been decided if or when the spark.mllib package will be
removed.  This cannot happen until spark.ml has complete feature parity and
has been separated from spark.mllib internally for a few releases, and it
will require a community vote and significant QA.
* You're correct that Evaluators are meant for model tuning.  IMO, multiple
metrics are more naturally handled by model/result summaries, though I
could see good arguments for separating the metric computation from
models.  This is an issue which has not yet been discussed properly.  There
have also been questions about Evaluators maintaining multiple metrics
along the way during model tuning (SPARK-18704).

I created a JIRA for discussing this further:
https://issues.apache.org/jira/browse/SPARK-19053

Thanks!
Joseph

On Thu, Dec 29, 2016 at 8:36 PM, Ilya Matiach <il...@microsoft.com> wrote:

> Hi ML/MLLib developers,
>
> 1.    I’m trying to add a weights column to ml spark evaluators
> (RegressionEvaluator, BinaryClassificationEvaluator,
> MutliclassClassificationEvaluator) that use mllib metrics and I have a
> few questions (JIRA
>
> 2.    SPARK-18693 <https://issues.apache.org/jira/browse/SPARK-18693>).
> I didn’t see any similar question on the forums or stackoverflow.
>
> Moving forward, will we keep mllib metrics (RegressionMetrics,
> MulticlassMetrics, BinaryClassificationMetrics) as something separate to
> the evaluators, or will we remove them when mllib is removed in spark 3.0?
>
The mllib metrics seem very useful because they are able to compute/expose
> many metrics on one dataset, whereas with the evaluators it is not
> performant to re-evaluate the entire dataset for a single different metric.
>
> For example, if I calculate the RMSE and then MSE using the ML
> RegressionEvaluator, I will be redoing most of the work twice, so the ML
> api doesn’t make sense to use in this scenario.
>
Also, the ml evaluators expose a lot fewer metrics than the mllib metrics
> classes, so it seems like the ml evaluators are not at parity with the
> mllib metrics classes.
>
> I can see how the ml evaluators are useful in CrossValidator, but for
> exploring all metrics from a scored dataset it doesn’t really make sense to
> use them.
>
> From the viewpoint of exploring all metrics for a scored model, does this
> mean that the mllib metrics classes should be moved to ml?
>
That would solve my issue if that is what is planned in the future.
> However, that doesn’t make sense to me, because it may cause some confusion
> for ml users to see metrics and evaluators classes.
>
>
>
> Instead, it seems like the ml evaluators need to be changed at the api
> layer to:
>
>    1. Allow the user to either retrieve a single value
>    2. Allow the user to retrieve all metrics or a set of metrics
>
> One possibility would be to overload evaluate so that we would have
> something like:
>
>
>
> override def evaluate(dataset: Dataset[_]): Double
>
> override def evaluate(dataset: Dataset[_], metrics:Array[String]):
> Dataset[_]
>
>
>
> But for some metrics like confusion matrix you couldn’t really fit the
> data into the result of the second api in addition to the single-value
> metrics.
>
> The format of the mllib metrics classes was much more convenient, as you
> could retrieve them directly.
>
> Following this line of thought, maybe the APIs could be:
>
>
>
> override def evaluate(dataset: Dataset[_]): Double
>
> def evaluateMetrics(dataset: Dataset[_]): RegressionEvaluation (or
> classification/multiclass etc)
>
>
>
> where the evaluation class returned will have very similar fields to the
> corresponding mllib RegressionMetrics class that can be called by the user.
>
>
>
> Any thoughts/ideas about spark ml evaluators/mllib metrics apis, coding
> suggestions for the api proposed, or a general roadmap would be really
> appreciated.
>
>
>
> Thank you, Ilya
>



-- 

Joseph Bradley

Software Engineer - Machine Learning

Databricks, Inc.

[image: http://databricks.com] <http://databricks.com/>