[jira] [Commented] (MADLIB-1169) Change how cross validation stats are reported and improve user docs

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MADLIB-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280834#comment-16280834 ] 

ASF GitHub Bot commented on MADLIB-1169:
----------------------------------------

GitHub user njayaram2 opened a pull request:

    https://github.com/apache/madlib/pull/210

    Regularized Regression: Change cross validation stats

    JIRA:MADLIB-1169
    
    Cross Validation seems to be supported by Elastic Net, SVM, and
    Decision Trees. If a module is run with cross validation optimization
    params, the output table corresponding to it displays `mean` and
    `std` of the negative loss error for each permutation of the CV
    params.
    - This commit changes column names: `mean`->`mean_neg_loss` and
    `std`->`std_neg_loss`.
    - CV now uses negative Root Mean Squared Error, instead
    of the negative Mean Squared Error.
    - Update Elastic Net user docs to reflect these changes.
    
    Closes #210

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/njayaram2/madlib improve/cross_validation_output

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/madlib/pull/210.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #210
    
----
commit ad41936c1a80e06cac6109690c15d7ad510c664d
Author: Nandish Jayaram <nj...@apache.org>
Date:   2017-12-06T19:58:46Z

    Regularized Regression: Change cross validation stats
    
    JIRA:MADLIB-1169
    
    Cross Validation seems to be supported by Elastic Net, SVM, and
    Decision Trees. If a module is run with cross validation optimization
    params, the output table corresponding to it displays `mean` and
    `std` of the negative loss error for each permutation of the CV
    params.
    - This commit changes column names: `mean`->`mean_neg_loss` and
    `std`->`std_neg_loss`.
    - CV now uses negative Root Mean Squared Error, instead
    of the negative Mean Squared Error.
    - Update Elastic Net user docs to reflect these changes.
    
    Closes #210

----


> Change how cross validation stats are reported and improve user docs 
> ---------------------------------------------------------------------
>
>                 Key: MADLIB-1169
>                 URL: https://issues.apache.org/jira/browse/MADLIB-1169
>             Project: Apache MADlib
>          Issue Type: Improvement
>          Components: Module: Regularized Regression
>            Reporter: Frank McQuillan
>            Priority: Minor
>             Fix For: v1.13
>
>
> Context
> Currently in cross validation, e.g.. for elastic net
> http://madlib.apache.org/docs/latest/group__grp__elasticnet.html
> CV stats are reported like:
> alpha | lambda_value |        mean         |     std
> ------+--------------+---------------------+--------------------
>     0 |       100000 | -1.41777698585e+110 | 1.80536123195e+110
>   0.1 |       100000 | -1.19953054719e+107 | 1.72846143163e+107
>     1 |       100000 |      -4175743937.91 |      2485189261.38
> etc.
> Here the "mean" col is the negative of the loss, which is a sort of accuracy, so the col header is not explanatory.
> Story
> As a MADlib developer, I want to make it clear what CV reported stats are, so that users are not confused as to what they mean.
> Acceptance
> 1) Change the calculation to report rmse from mse so that it is a smaller number that relates to the magnitude of the data.  We can still report as negative, but just use rmse.
> 2) Rename the columns as "mean_neg_loss: and "std_neg_loss"
> 3) Improve the user docs to explain the col means for regression and classification.
> 4) Update any IC or functional/Tinc tests that are affected.
> 5) Update the example in 
> http://madlib.apache.org/docs/latest/group__grp__elasticnet.html
> and also find any other modules where that need to be updated (SVM?)
> 6) Check the log_likelihood value reported in EN.  Is it really a log likelihood and is it reported correctly?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)