[jira] [Comment Edited] (SPARK-1227) Diagnostics for Classification&Regression

["Martin Jaggi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204713#comment-14204713 ] 

Martin Jaggi edited comment on SPARK-1227 at 11/10/14 12:33 PM:
----------------------------------------------------------------

actually this is still relevant, as looking at the training objective value is the only way to tell if a chosen model has been properly trained or not. without this, the user will not know if a resulting bad test error should be blamed on poor training of a good model, or on choosing a wrong model (e.g. wrong regularization parameter). both happens often. or imagine a deep net, where the same question is even harder to tell apart.

now that MLlib starts offering different training algorithms for the same models (e.g. SGD and L-BFGS), and also different ways of distributing training, the training objective would definitely be useful compare algorithms. (or also when comparing different step-size regimes, such as here:  https://issues.apache.org/jira/browse/SPARK-3942  )

maybe the nicest way around this would be to provide a nice *benchmarking suite* for regression and classification, which could be used to judge different algorithms in this respect, also if new contributed algorithms need to be compared for efficiency. a benchmarking suite should include a small selection of standard datasets for both sparse and dense data.

this is also related to the currently not so nice way of passing around the regularizer values (which are part of the training optimization objective) through the updater function, which is currently quite different in SGD as compared to L-BFGS, see the issue here:
https://issues.apache.org/jira/browse/SPARK-2505
and the spark L-BFBS implementation:
https://github.com/apache/spark/pull/353

irrespective of training error, the classifier methods would also benefit from adding the test accuracy percentage as a function (see current code examples, where this still has to be calculated manually, as it's not implemented yet in {{BinaryClassificationMetrics}}.


was (Author: jaggi):
actually this is still relevant, as looking at the training objective value is the only way to tell if a chosen model has been properly trained or not. without this, the user will not know if a resulting bad test error should be blamed on poor training of a good model, or on choosing a wrong model (e.g. wrong regularization parameter). both happens often. or imagine a deep net, where the same question is even harder to tell apart.

now that MLlib starts offering different training algorithms for the same models (e.g. SGD and L-BFGS), and also different ways of distributing training, the training objective would definitely be useful compare algorithms. (or also when comparing different step-seize regimes, such as here:  https://issues.apache.org/jira/browse/SPARK-3942  )

maybe the nicest way around this would be to provide a nice *benchmarking suite* for regression and classification, which could be used to judge different algorithms in this respect, also if new contributed algorithms need to be compared for efficiency.

this is also related to the currently not so nice way of passing around the regularizer values (which are part of the training optimization objective) through the updater function, which is currently quite different in SGD as compared to L-BFGS, see the issue here:
https://issues.apache.org/jira/browse/SPARK-2505
and the spark L-BFBS implementation:
https://github.com/apache/spark/pull/353

irrespective of training error, the classifier methods would also benefit from adding the test accuracy percentage as a function (see current code examples, where this still has to be calculated manually, as it's not implemented yet in {{BinaryClassificationMetrics}}.

> Diagnostics for Classification&Regression
> -----------------------------------------
>
>                 Key: SPARK-1227
>                 URL: https://issues.apache.org/jira/browse/SPARK-1227
>             Project: Spark
>          Issue Type: Improvement
>          Components: MLlib
>            Reporter: Martin Jaggi
>            Assignee: Martin Jaggi
>
> Currently, the attained objective function is not computed (for efficiency reasons, as one evaluation requires one full pass through the data).
> For diagnostics and comparing different algorithms, we should however provide this as a separate function (one MR).
> Doing this requires the loss and regularizer functions themselves, not only their gradients (which are currently in the Gradient class). How about adding the new function directly on the corresponding models in classification/* and regression/* ? Any thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org