You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/04/16 09:56:49 UTC
[GitHub] [incubator-mxnet] acphile opened a new pull request #18083: Changes
to mxnet.metric
acphile opened a new pull request #18083: Changes to mxnet.metric
URL: https://github.com/apache/incubator-mxnet/pull/18083
## Description ##
change based on https://github.com/apache/incubator-mxnet/issues/18046
1. make improvements in metric
a. improve Class MAE (and MSE, RMSE)
b. improve Class _BinaryClassification
c. improve Class TopKAccuracy
d. add Class MeanCosineSimilarity
e. add Class MeanPairwiseDistance
2. move mxnet.metric to mxnet.gluon.metric
## Checklist ##
### Essentials ###
Please feel free to remove inapplicable items for your PR.
- [ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes)
- [ ] Changes are complete (i.e. I finished coding on this PR)
- [ ] All changes have test coverage:
- Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
- Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
- Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
- [ ] Code is well-documented:
- For user-facing API changes, API doc string has been updated.
- For new C++ functions in header files, their functionalities and arguments are documented.
- For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
- Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
- [ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
### Changes ###
- [ ] Feature1, tests, (and when applicable, API doc)
- [ ] Feature2, tests, (and when applicable, API doc)
## Comments ##
- If this change is a backward incompatible change, why must this change be made.
- Interesting edge cases to note here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] acphile commented on a change in pull request
#18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on a change in pull request #18083: Changes to mxnet.metric
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r410213003
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -777,16 +815,113 @@ class F1(EvalMetric):
--------
>>> predicts = [mx.nd.array([[0.3, 0.7], [0., 1.], [0.4, 0.6]])]
>>> labels = [mx.nd.array([0., 1., 1.])]
- >>> f1 = mx.metric.F1()
+ >>> f1 = mx.gluon.metric.F1()
>>> f1.update(preds = predicts, labels = labels)
>>> print f1.get()
('f1', 0.8)
"""
def __init__(self, name='f1',
- output_names=None, label_names=None, average="macro"):
+ output_names=None, label_names=None, threshold=0.5, average="macro"):
+ self.average = average
+ self.metrics = _BinaryClassificationMetrics(threshold=threshold)
+ EvalMetric.__init__(self, name=name,
+ output_names=output_names, label_names=label_names,
+ has_global_stats=True)
+
+ def update(self, labels, preds):
+ """Updates the internal evaluation result.
+
+ Parameters
+ ----------
+ labels : list of `NDArray`
+ The labels of the data.
+
+ preds : list of `NDArray`
+ Predicted values.
+ """
+ labels, preds = check_label_shapes(labels, preds, True)
+
+ for label, pred in zip(labels, preds):
+ self.metrics.update_binary_stats(label, pred)
+
+ if self.average == "macro":
Review comment:
Averaging F1 per batch here previously existed in metric.py before I made changes. This calculation also exists in MAE, MSE, RMSE, and PearsonCorrelation. Should I remove all of them accordingly? For the average "macro" in sklearn, it seems used in calculating F1 score for multiclass/multilabel targets. But currently our F1 only supports binary classification. I think I need to make extensions for F1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] sxjscience commented on a change in pull request
#18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
sxjscience commented on a change in pull request #18083: Changes to mxnet.metric
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r409659282
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -1130,12 +1339,18 @@ def update(self, labels, preds):
label = label.reshape(label.shape[0], 1)
if len(pred.shape) == 1:
pred = pred.reshape(pred.shape[0], 1)
-
- mae = numpy.abs(label - pred).mean()
+
+ if self.average == "macro":
Review comment:
It's very strange to have `macro` + MAE. See scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html#sklearn.metrics.mean_absolute_error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] mxnet-bot commented on issue #18083: Changes to
mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on issue #18083: Changes to mxnet.metric
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-614544916
Hey @acphile , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [centos-cpu, sanity, miscellaneous, centos-gpu, windows-cpu, website, unix-gpu, edge, windows-gpu, clang, unix-cpu]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] mseth10 commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mseth10 commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-628494904
@acphile this PR fails nightly CD while running nightly python unit tests. The following tests fail:
test_mcc, test_multilabel_f1, test_binary_f1
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fmxnet-cd-release-job/detail/mxnet-cd-release-job/1119/pipeline/329
We'll need to revert this PR and fix the failures before re-merging it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on a change in pull request
#18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
sxjscience commented on a change in pull request #18083: Changes to mxnet.metric
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r410271991
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -777,16 +815,113 @@ class F1(EvalMetric):
--------
>>> predicts = [mx.nd.array([[0.3, 0.7], [0., 1.], [0.4, 0.6]])]
>>> labels = [mx.nd.array([0., 1., 1.])]
- >>> f1 = mx.metric.F1()
+ >>> f1 = mx.gluon.metric.F1()
>>> f1.update(preds = predicts, labels = labels)
>>> print f1.get()
('f1', 0.8)
"""
def __init__(self, name='f1',
- output_names=None, label_names=None, average="macro"):
+ output_names=None, label_names=None, threshold=0.5, average="macro"):
+ self.average = average
+ self.metrics = _BinaryClassificationMetrics(threshold=threshold)
+ EvalMetric.__init__(self, name=name,
+ output_names=output_names, label_names=label_names,
+ has_global_stats=True)
+
+ def update(self, labels, preds):
+ """Updates the internal evaluation result.
+
+ Parameters
+ ----------
+ labels : list of `NDArray`
+ The labels of the data.
+
+ preds : list of `NDArray`
+ Predicted values.
+ """
+ labels, preds = check_label_shapes(labels, preds, True)
+
+ for label, pred in zip(labels, preds):
+ self.metrics.update_binary_stats(label, pred)
+
+ if self.average == "macro":
Review comment:
Let's remove it and make it similar to sklearn. This is in fact the reason why I never use the metric class in MXNet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r421715173
##########
File path: tests/python/tensorrt/test_cvnets.py
##########
@@ -29,7 +28,12 @@
def get_classif_model(model_name, use_tensorrt, ctx=mx.gpu(0), batch_size=128):
mx.contrib.tensorrt.set_use_fp16(False)
h, w = 32, 32
- net = gluoncv.model_zoo.get_model(model_name, pretrained=True)
+ model_url = "https://raw.githubusercontent.com/dmlc/web-data/master/gluoncv/models/"
Review comment:
Please don't hardcode `master` in the URL here. The repository may change and will then break the CI. Instead, use the commit ID: `https://raw.githubusercontent.com/dmlc/web-data/221ce5b7c6d5b0777a1e3471f7f03ff98da90a0a/gluoncv/models/"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r421715173
##########
File path: tests/python/tensorrt/test_cvnets.py
##########
@@ -29,7 +28,12 @@
def get_classif_model(model_name, use_tensorrt, ctx=mx.gpu(0), batch_size=128):
mx.contrib.tensorrt.set_use_fp16(False)
h, w = 32, 32
- net = gluoncv.model_zoo.get_model(model_name, pretrained=True)
+ model_url = "https://raw.githubusercontent.com/dmlc/web-data/master/gluoncv/models/"
Review comment:
Please don't hardcode `master` in the URL here. The repository may change and will then break the CI. Instead, use the commit ID: `https://raw.githubusercontent.com/dmlc/web-data/221ce5b7c6d5b0777a1e3471f7f03ff98da90a0a/gluoncv/models`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r412625767
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -619,90 +583,100 @@ def update_binary_stats(self, label, pred):
"""
pred = pred.asnumpy()
label = label.asnumpy().astype('int32')
- pred_label = numpy.argmax(pred, axis=1)
-
- check_label_shapes(label, pred)
- if len(numpy.unique(label)) > 2:
- raise ValueError("%s currently only supports binary classification."
- % self.__class__.__name__)
+ if self.class_type == "binary":
+ self._set(1)
+ if len(numpy.unique(label)) > 2:
+ raise ValueError("Wrong label for binary classification.")
+ if pred.shape == label.shape:
+ pass
+ elif pred.shape[-1] > 2:
+ raise ValueError("The shape of prediction {} is wrong for binary classification.".format(pred.shape))
+ elif pred.shape[-1] == 2:
+ pred = pred.reshape(-1, 2)[:, 1]
+ pred_label = predict_with_threshold(pred, self.threshold).flat
+ label = label.flat
+
+ elif self.class_type == "multiclass":
+ num = pred.shape[-1]
+ self._set(num)
+ assert label.max() < num, "pred contains fewer classes than label!"
+ pred_label = one_hot(pred.argmax(axis=-1).reshape(-1), num)
+ label = one_hot(label.reshape(-1), num)
+
+ elif self.class_type == "multilabel":
+ num = pred.shape[-1]
+ self._set(num)
+ assert pred.shape == label.shape, "The shape of label should be same as that of prediction for multilabel classification."
+ pred_label = predict_with_threshold(pred, self.threshold).reshape(-1, num)
+ label = label.reshape(-1, num)
+ else:
+ raise ValueError("Wrong class_type {}! Only supports ['binary', 'multiclass', 'multilabel']".format(self.class_type))
+
+ check_label_shapes(label, pred_label)
+
pred_true = (pred_label == 1)
pred_false = 1 - pred_true
label_true = (label == 1)
label_false = 1 - label_true
- true_pos = (pred_true * label_true).sum()
- false_pos = (pred_true * label_false).sum()
- false_neg = (pred_false * label_true).sum()
- true_neg = (pred_false * label_false).sum()
+ true_pos = (pred_true * label_true).sum(0)
+ false_pos = (pred_true * label_false).sum(0)
+ false_neg = (pred_false * label_true).sum(0)
+ true_neg = (pred_false * label_false).sum(0)
self.true_positives += true_pos
- self.global_true_positives += true_pos
self.false_positives += false_pos
- self.global_false_positives += false_pos
self.false_negatives += false_neg
- self.global_false_negatives += false_neg
self.true_negatives += true_neg
- self.global_true_negatives += true_neg
@property
def precision(self):
- if self.true_positives + self.false_positives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_positives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_positives, 1e-12)
else:
return 0.
@property
def global_precision(self):
- if self.global_true_positives + self.global_false_positives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_positives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_positives.sum(), 1e-12)
else:
return 0.
-
+
@property
def recall(self):
- if self.true_positives + self.false_negatives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_negatives, 1e-12)
else:
return 0.
@property
def global_recall(self):
- if self.global_true_positives + self.global_false_negatives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_negatives.sum(), 1e-12)
else:
return 0.
-
+
@property
def fscore(self):
- if self.precision + self.recall > 0:
- return 2 * self.precision * self.recall / (self.precision + self.recall)
- else:
- return 0.
+ return (1 + self.beta ** 2) * self.precision * self.recall / numpy.maximum(self.beta ** 2 * self.precision + self.recall, 1e-12)
@property
def global_fscore(self):
Review comment:
Would it make sense to adjust the name?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-625042934
@mxnet-bot run ci [unix-cpu, windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r414924090
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -617,92 +585,102 @@ def update_binary_stats(self, label, pred):
pred : `NDArray`
Predicted values.
"""
- pred = pred.asnumpy()
- label = label.asnumpy().astype('int32')
- pred_label = numpy.argmax(pred, axis=1)
-
- check_label_shapes(label, pred)
- if len(numpy.unique(label)) > 2:
- raise ValueError("%s currently only supports binary classification."
- % self.__class__.__name__)
+ pred = pred.as_np_ndarray()
+ label = label.as_np_ndarray().astype('int32')
+ if self.class_type == "binary":
+ self._set(1)
+ if len(numpy.unique(label)) > 2:
Review comment:
https://github.com/apache/incubator-mxnet/blob/83b51703ed354f41024423f140de38df2ba22d50/src/imperative/imperative.cc#L123-L127
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-622272318
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r414921455
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -617,92 +585,102 @@ def update_binary_stats(self, label, pred):
pred : `NDArray`
Predicted values.
"""
- pred = pred.asnumpy()
- label = label.asnumpy().astype('int32')
- pred_label = numpy.argmax(pred, axis=1)
-
- check_label_shapes(label, pred)
- if len(numpy.unique(label)) > 2:
- raise ValueError("%s currently only supports binary classification."
- % self.__class__.__name__)
+ pred = pred.as_np_ndarray()
+ label = label.as_np_ndarray().astype('int32')
+ if self.class_type == "binary":
+ self._set(1)
+ if len(numpy.unique(label)) > 2:
Review comment:
This will trigger synchronization (as we need to wait for the result of the np.unique operator). Could we make error checking that triggers synchronization optional?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-626196824
Jenkins CI successfully triggered : [unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-628746204
The reason it fails is that the CI check in this PR was run too long ago. I should have restarted it before merging the PR. Meantime master changed and caused some additional changes to be necessary. They are in https://github.com/apache/incubator-mxnet/pull/18312/files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on a change in pull request
#18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
sxjscience commented on a change in pull request #18083: Changes to mxnet.metric
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r409657973
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -777,16 +815,113 @@ class F1(EvalMetric):
--------
>>> predicts = [mx.nd.array([[0.3, 0.7], [0., 1.], [0.4, 0.6]])]
>>> labels = [mx.nd.array([0., 1., 1.])]
- >>> f1 = mx.metric.F1()
+ >>> f1 = mx.gluon.metric.F1()
>>> f1.update(preds = predicts, labels = labels)
>>> print f1.get()
('f1', 0.8)
"""
def __init__(self, name='f1',
- output_names=None, label_names=None, average="macro"):
+ output_names=None, label_names=None, threshold=0.5, average="macro"):
+ self.average = average
+ self.metrics = _BinaryClassificationMetrics(threshold=threshold)
+ EvalMetric.__init__(self, name=name,
+ output_names=output_names, label_names=label_names,
+ has_global_stats=True)
+
+ def update(self, labels, preds):
+ """Updates the internal evaluation result.
+
+ Parameters
+ ----------
+ labels : list of `NDArray`
+ The labels of the data.
+
+ preds : list of `NDArray`
+ Predicted values.
+ """
+ labels, preds = check_label_shapes(labels, preds, True)
+
+ for label, pred in zip(labels, preds):
+ self.metrics.update_binary_stats(label, pred)
+
+ if self.average == "macro":
Review comment:
In fact, `macro averaging` + F1 does not mean to average the F1 of each batch. I think we should revise it to be the same as https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621663975
It's currently broken. https://github.com/apache/incubator-mxnet/issues/17840 Needs to be fixed before it can be enabled on CI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621401850
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r412613930
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -24,9 +24,9 @@
import numpy
Review comment:
Instead of using numpy, we can use `mxnet.numpy` as it runs asynchronously and has GPU support.
The summary states of a metric should be stored on CPU, but if for example data and label are on GPU and input to the metric, we can calculate the sufficient statistics on GPU
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r422279290
##########
File path: tests/python/train/test_mlp.py
##########
@@ -37,8 +37,9 @@ def test_mlp(tmpdir):
def accuracy(label, pred):
py = np.argmax(pred, axis=1)
- return np.sum(py == label) / float(label.size)
-
+ return np.sum(py == label.astype(py)) / float(label.size)
+ # currently mxnet.numpy (which used in gluon.metric) did not support "==" between different types
Review comment:
Reference https://github.com/apache/incubator-mxnet/issues/18137
##########
File path: tests/python/unittest/test_metric.py
##########
@@ -391,19 +367,23 @@ def test_single_array_input():
pred = mx.nd.array([[1,2,3,4]])
label = pred + 0.1
- mse = mx.metric.create('mse')
+ mse = mx.gluon.metric.create('mse')
mse.update(label, pred)
_, mse_res = mse.get()
np.testing.assert_almost_equal(mse_res, 0.01)
- mae = mx.metric.create('mae')
+ mae = mx.gluon.metric.create('mae')
mae.update(label, pred)
mae.get()
_, mae_res = mae.get()
np.testing.assert_almost_equal(mae_res, 0.1)
- rmse = mx.metric.create('rmse')
+ rmse = mx.gluon.metric.create('rmse')
rmse.update(label, pred)
rmse.get()
_, rmse_res = rmse.get()
np.testing.assert_almost_equal(rmse_res, 0.1)
+
+if __name__ == '__main__':
+ import nose
+ nose.runmodule()
Review comment:
This can be deleted
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-625042951
Jenkins CI successfully triggered : [unix-cpu, windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r412605078
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -619,90 +583,100 @@ def update_binary_stats(self, label, pred):
"""
pred = pred.asnumpy()
label = label.asnumpy().astype('int32')
- pred_label = numpy.argmax(pred, axis=1)
-
- check_label_shapes(label, pred)
- if len(numpy.unique(label)) > 2:
- raise ValueError("%s currently only supports binary classification."
- % self.__class__.__name__)
+ if self.class_type == "binary":
+ self._set(1)
+ if len(numpy.unique(label)) > 2:
+ raise ValueError("Wrong label for binary classification.")
+ if pred.shape == label.shape:
+ pass
+ elif pred.shape[-1] > 2:
+ raise ValueError("The shape of prediction {} is wrong for binary classification.".format(pred.shape))
+ elif pred.shape[-1] == 2:
+ pred = pred.reshape(-1, 2)[:, 1]
+ pred_label = predict_with_threshold(pred, self.threshold).flat
+ label = label.flat
+
+ elif self.class_type == "multiclass":
+ num = pred.shape[-1]
+ self._set(num)
+ assert label.max() < num, "pred contains fewer classes than label!"
+ pred_label = one_hot(pred.argmax(axis=-1).reshape(-1), num)
+ label = one_hot(label.reshape(-1), num)
+
+ elif self.class_type == "multilabel":
+ num = pred.shape[-1]
+ self._set(num)
+ assert pred.shape == label.shape, "The shape of label should be same as that of prediction for multilabel classification."
+ pred_label = predict_with_threshold(pred, self.threshold).reshape(-1, num)
+ label = label.reshape(-1, num)
+ else:
+ raise ValueError("Wrong class_type {}! Only supports ['binary', 'multiclass', 'multilabel']".format(self.class_type))
+
+ check_label_shapes(label, pred_label)
+
pred_true = (pred_label == 1)
pred_false = 1 - pred_true
label_true = (label == 1)
label_false = 1 - label_true
- true_pos = (pred_true * label_true).sum()
- false_pos = (pred_true * label_false).sum()
- false_neg = (pred_false * label_true).sum()
- true_neg = (pred_false * label_false).sum()
+ true_pos = (pred_true * label_true).sum(0)
+ false_pos = (pred_true * label_false).sum(0)
+ false_neg = (pred_false * label_true).sum(0)
+ true_neg = (pred_false * label_false).sum(0)
self.true_positives += true_pos
- self.global_true_positives += true_pos
self.false_positives += false_pos
- self.global_false_positives += false_pos
self.false_negatives += false_neg
- self.global_false_negatives += false_neg
self.true_negatives += true_neg
- self.global_true_negatives += true_neg
@property
def precision(self):
- if self.true_positives + self.false_positives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_positives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_positives, 1e-12)
else:
return 0.
@property
def global_precision(self):
- if self.global_true_positives + self.global_false_positives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_positives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_positives.sum(), 1e-12)
else:
return 0.
-
+
@property
def recall(self):
- if self.true_positives + self.false_negatives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_negatives, 1e-12)
else:
return 0.
@property
def global_recall(self):
- if self.global_true_positives + self.global_false_negatives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_negatives.sum(), 1e-12)
else:
return 0.
-
+
@property
def fscore(self):
- if self.precision + self.recall > 0:
- return 2 * self.precision * self.recall / (self.precision + self.recall)
- else:
- return 0.
+ return (1 + self.beta ** 2) * self.precision * self.recall / numpy.maximum(self.beta ** 2 * self.precision + self.recall, 1e-12)
@property
def global_fscore(self):
Review comment:
This method should be removed as you dropped the global states?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r412624461
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -619,90 +583,100 @@ def update_binary_stats(self, label, pred):
"""
pred = pred.asnumpy()
label = label.asnumpy().astype('int32')
- pred_label = numpy.argmax(pred, axis=1)
-
- check_label_shapes(label, pred)
- if len(numpy.unique(label)) > 2:
- raise ValueError("%s currently only supports binary classification."
- % self.__class__.__name__)
+ if self.class_type == "binary":
+ self._set(1)
+ if len(numpy.unique(label)) > 2:
+ raise ValueError("Wrong label for binary classification.")
+ if pred.shape == label.shape:
+ pass
+ elif pred.shape[-1] > 2:
+ raise ValueError("The shape of prediction {} is wrong for binary classification.".format(pred.shape))
+ elif pred.shape[-1] == 2:
+ pred = pred.reshape(-1, 2)[:, 1]
+ pred_label = predict_with_threshold(pred, self.threshold).flat
+ label = label.flat
+
+ elif self.class_type == "multiclass":
+ num = pred.shape[-1]
+ self._set(num)
+ assert label.max() < num, "pred contains fewer classes than label!"
+ pred_label = one_hot(pred.argmax(axis=-1).reshape(-1), num)
+ label = one_hot(label.reshape(-1), num)
+
+ elif self.class_type == "multilabel":
+ num = pred.shape[-1]
+ self._set(num)
+ assert pred.shape == label.shape, "The shape of label should be same as that of prediction for multilabel classification."
+ pred_label = predict_with_threshold(pred, self.threshold).reshape(-1, num)
+ label = label.reshape(-1, num)
+ else:
+ raise ValueError("Wrong class_type {}! Only supports ['binary', 'multiclass', 'multilabel']".format(self.class_type))
+
+ check_label_shapes(label, pred_label)
+
pred_true = (pred_label == 1)
pred_false = 1 - pred_true
label_true = (label == 1)
label_false = 1 - label_true
- true_pos = (pred_true * label_true).sum()
- false_pos = (pred_true * label_false).sum()
- false_neg = (pred_false * label_true).sum()
- true_neg = (pred_false * label_false).sum()
+ true_pos = (pred_true * label_true).sum(0)
+ false_pos = (pred_true * label_false).sum(0)
+ false_neg = (pred_false * label_true).sum(0)
+ true_neg = (pred_false * label_false).sum(0)
self.true_positives += true_pos
- self.global_true_positives += true_pos
self.false_positives += false_pos
- self.global_false_positives += false_pos
self.false_negatives += false_neg
- self.global_false_negatives += false_neg
self.true_negatives += true_neg
- self.global_true_negatives += true_neg
@property
def precision(self):
- if self.true_positives + self.false_positives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_positives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_positives, 1e-12)
else:
return 0.
@property
def global_precision(self):
- if self.global_true_positives + self.global_false_positives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_positives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_positives.sum(), 1e-12)
else:
return 0.
-
+
@property
def recall(self):
- if self.true_positives + self.false_negatives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_negatives, 1e-12)
else:
return 0.
@property
def global_recall(self):
- if self.global_true_positives + self.global_false_negatives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_negatives.sum(), 1e-12)
else:
return 0.
-
+
@property
def fscore(self):
- if self.precision + self.recall > 0:
- return 2 * self.precision * self.recall / (self.precision + self.recall)
- else:
- return 0.
+ return (1 + self.beta ** 2) * self.precision * self.recall / numpy.maximum(self.beta ** 2 * self.precision + self.recall, 1e-12)
@property
def global_fscore(self):
Review comment:
This method actually refers to the micro calculation for F1 and it is not related to original global support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-620311220
Jenkins CI successfully triggered : [centos-gpu, centos-cpu, windows-gpu, unix-cpu, sanity, unix-gpu, windows-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621401969
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-620311153
@mxnet-bot run ci [centos-cpu, sanity, centos-gpu, windows-cpu, unix-gpu, windows-gpu, unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621664862
Please disable it in a separate PR then. I'd like to have a better understanding of what exactly is broken , but not have that discussion in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-625690567
Jenkins CI successfully triggered : [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621665699
You can go ahead and try reproduce the issue locally:
The reproducer is available since more than a month at https://github.com/apache/incubator-mxnet/issues/17886#issue-585864921
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621661999
Why are we removing tvm op?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621667142
You're very welcome to resolve this issue and create a separate PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu merged pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu merged pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mseth10 edited a comment on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mseth10 edited a comment on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-628494904
@acphile this PR fails nightly CD while running nightly python unit tests. The following tests fail:
test_mcc, test_multilabel_f1, test_binary_f1
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-mxnet-cd%2Fmxnet-cd-release-job/detail/mxnet-cd-release-job/1119/pipeline/329
We'll need to revert this PR and fix the failures before re-merging it. Here's the link to revert PR: https://github.com/apache/incubator-mxnet/pull/18318
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-622272347
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-625690495
@mxnet-bot run ci [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621666446
No, it's unrelated and should be a separated and isolated PR. Each PR should serve one propose. That way, we can focus discussions, have single purpose commits and also allow reverting
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621669161
Let's disable it in https://github.com/apache/incubator-mxnet/pull/18204
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621402024
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621587674
@mxnet-bot run ci [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu removed a comment on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu removed a comment on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621667142
You're very welcome to resolve this issue and create a separate PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621401895
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621587701
Jenkins CI successfully triggered : [unix-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on a change in pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on a change in pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#discussion_r412635373
##########
File path: python/mxnet/gluon/metric.py
##########
@@ -619,90 +583,100 @@ def update_binary_stats(self, label, pred):
"""
pred = pred.asnumpy()
label = label.asnumpy().astype('int32')
- pred_label = numpy.argmax(pred, axis=1)
-
- check_label_shapes(label, pred)
- if len(numpy.unique(label)) > 2:
- raise ValueError("%s currently only supports binary classification."
- % self.__class__.__name__)
+ if self.class_type == "binary":
+ self._set(1)
+ if len(numpy.unique(label)) > 2:
+ raise ValueError("Wrong label for binary classification.")
+ if pred.shape == label.shape:
+ pass
+ elif pred.shape[-1] > 2:
+ raise ValueError("The shape of prediction {} is wrong for binary classification.".format(pred.shape))
+ elif pred.shape[-1] == 2:
+ pred = pred.reshape(-1, 2)[:, 1]
+ pred_label = predict_with_threshold(pred, self.threshold).flat
+ label = label.flat
+
+ elif self.class_type == "multiclass":
+ num = pred.shape[-1]
+ self._set(num)
+ assert label.max() < num, "pred contains fewer classes than label!"
+ pred_label = one_hot(pred.argmax(axis=-1).reshape(-1), num)
+ label = one_hot(label.reshape(-1), num)
+
+ elif self.class_type == "multilabel":
+ num = pred.shape[-1]
+ self._set(num)
+ assert pred.shape == label.shape, "The shape of label should be same as that of prediction for multilabel classification."
+ pred_label = predict_with_threshold(pred, self.threshold).reshape(-1, num)
+ label = label.reshape(-1, num)
+ else:
+ raise ValueError("Wrong class_type {}! Only supports ['binary', 'multiclass', 'multilabel']".format(self.class_type))
+
+ check_label_shapes(label, pred_label)
+
pred_true = (pred_label == 1)
pred_false = 1 - pred_true
label_true = (label == 1)
label_false = 1 - label_true
- true_pos = (pred_true * label_true).sum()
- false_pos = (pred_true * label_false).sum()
- false_neg = (pred_false * label_true).sum()
- true_neg = (pred_false * label_false).sum()
+ true_pos = (pred_true * label_true).sum(0)
+ false_pos = (pred_true * label_false).sum(0)
+ false_neg = (pred_false * label_true).sum(0)
+ true_neg = (pred_false * label_false).sum(0)
self.true_positives += true_pos
- self.global_true_positives += true_pos
self.false_positives += false_pos
- self.global_false_positives += false_pos
self.false_negatives += false_neg
- self.global_false_negatives += false_neg
self.true_negatives += true_neg
- self.global_true_negatives += true_neg
@property
def precision(self):
- if self.true_positives + self.false_positives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_positives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_positives, 1e-12)
else:
return 0.
@property
def global_precision(self):
- if self.global_true_positives + self.global_false_positives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_positives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_positives.sum(), 1e-12)
else:
return 0.
-
+
@property
def recall(self):
- if self.true_positives + self.false_negatives > 0:
- return float(self.true_positives) / (self.true_positives + self.false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives / numpy.maximum(self.true_positives + self.false_negatives, 1e-12)
else:
return 0.
@property
def global_recall(self):
- if self.global_true_positives + self.global_false_negatives > 0:
- return float(self.global_true_positives) / (self.global_true_positives + self.global_false_negatives)
+ if self.num_classes is not None:
+ return self.true_positives.sum() / numpy.maximum(self.true_positives.sum() + self.false_negatives.sum(), 1e-12)
else:
return 0.
-
+
@property
def fscore(self):
- if self.precision + self.recall > 0:
- return 2 * self.precision * self.recall / (self.precision + self.recall)
- else:
- return 0.
+ return (1 + self.beta ** 2) * self.precision * self.recall / numpy.maximum(self.beta ** 2 * self.precision + self.recall, 1e-12)
@property
def global_fscore(self):
Review comment:
I think it is ok to use global_fscore since it is in a private container class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-621665235
Let's disable it here, because it blocks this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] acphile commented on pull request #18083: Changes to mxnet.metric
Posted by GitBox <gi...@apache.org>.
acphile commented on pull request #18083:
URL: https://github.com/apache/incubator-mxnet/pull/18083#issuecomment-626196810
@mxnet-bot run ci [unix-cpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org