[GitHub] feevos opened a new issue #11949: Confusion in documentation/implementation of F1, MCC metrics

[GitBox <gi...@apache.org>]

feevos opened a new issue #11949: Confusion in documentation/implementation of F1, MCC metrics
URL: https://github.com/apache/incubator-mxnet/issues/11949
 
 
   Dear all,  
   
   I was looking at the ```mxnet.metric``` source [code](https://mxnet.incubator.apache.org/_modules/mxnet/metric.html#_BinaryClassificationMetrics) and [documentation](https://mxnet.incubator.apache.org/api/python/metric/metric.html) and I had a bit of confusion on the  F1 and MCC metrics. In particular, if one needs to provide a 1D or a 2D NDArray predictions.  I think, that since for the binary case either a 1D probability of True or a 2D probability of (False,True) predictions are equivalent, this should be unified in the code/documentation.  
   
   At present  one needs to provide a 2D NDArray of predictions and a 1D NDArray of labels. Yes, they are equivalent, however, in the 2D case, it **is not stated clearly which dimension represents the probability of ```True``` value**. I think this can produce confusion.   
   
   So if I have ```labels = np.array([1,0,0])``` and predictions, ```preds = np.array([0.8, 0.25, 0.15])```. In order to make any of the above metrics to work, I need to provide a 2D version of ```preds_2D = np.array(list(zip (1-preds, preds)))```, where the first element of the tuple corresponds to the ```False``` prediction, while the latter in the ```True``` . 
   
   ```Python 
   labels = np.array([1,0,0])
   preds = np.array([0.8, 0.25, 0.15])
   f1 = mxnet.metric.F1()
   f1.update(labels = nd.array(labels),preds = nd.array(preds))
   f1.get()
   # Output
   ('f1',1.0)
   ```
   
   I think it should be relatively easy to unify both approaches (1D and 2D) under the same operator. I've tracked down the problem in the class  ```_BinaryClassificationMetrics ```, 
   line  [534](https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/metric.py#L534). There one calls ```np.argmax(preds, axis=1)``` to get the label index. I think issue #9865 and this [discussion](https://discuss.mxnet.io/t/a-confusion-on-evaluation-metrics/662/3) are related.  
   
   Regards. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services