You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/11/16 22:01:27 UTC
[GitHub] [incubator-pinot] bryantachen opened a new pull request #6268: [TE] rest-api enhance implementation to calculate alert performance
bryantachen opened a new pull request #6268:
URL: https://github.com/apache/incubator-pinot/pull/6268
Enhanced Harley's implementation on calculating alert performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #6268: [TE] rest-api enhance implementation to calculate alert performance
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #6268:
URL: https://github.com/apache/incubator-pinot/pull/6268#discussion_r524749240
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/auth/ThirdEyeAuthFilter.java
##########
@@ -86,7 +86,7 @@ public void filter(ContainerRequestContext requestContext) {
return;
}
}
-
+
Review comment:
remove blank edits?
##########
File path: thirdeye/thirdeye-spi/src/main/java/org/apache/pinot/thirdeye/detection/performance/PerformanceMetrics.java
##########
@@ -0,0 +1,155 @@
+package org.apache.pinot.thirdeye.detection.performance;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.util.List;
+import org.apache.pinot.thirdeye.constant.AnomalyResultSource;
+import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
+
+
+/**
+ * The performance metrics of a detection based on the given anomalies
+ */
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class PerformanceMetrics {
+
+ @JsonProperty
+ private PerformanceMetric totalAnomalies;
+
+ @JsonProperty
+ private PerformanceMetric responseRate;
+
+ @JsonProperty
+ private PerformanceMetric precision;
+
+ @JsonProperty
+ private PerformanceMetric recall;
+
+ public PerformanceMetric getTotalAnomalies() { return totalAnomalies; }
+
+ public PerformanceMetric getResponseRate() { return responseRate; }
+
+ public PerformanceMetric getPrecision() { return precision; }
+
+ public PerformanceMetric getRecall() { return recall; }
+
+ /**
+ * Builder for performance metrics
+ */
+ public static class Builder {
+ private long respondedAnomalies, truePos, falsePos, falseNeg, numAnomalies = 0L;
+ private boolean includeTotalAnomalies, includeResponseRate, includePrecision, includeRecall = false;
+
+ private PerformanceMetric buildTotalAnomalies() {
+ PerformanceMetric totalAnomalies = new PerformanceMetric();
+ totalAnomalies.setValue((double) this.numAnomalies);
+ totalAnomalies.setType(PerformanceMetricType.COUNT);
+ return totalAnomalies;
+ }
+
+ private PerformanceMetric buildResponseRate() {
+ PerformanceMetric responseRate = new PerformanceMetric();
+ double rate = (double)this.respondedAnomalies / this.numAnomalies * 100;
+ responseRate.setValue(rate);
+ responseRate.setType(PerformanceMetricType.PERCENT);
+ return responseRate;
+ }
+
+ private PerformanceMetric buildPrecision() {
+ PerformanceMetric precision = new PerformanceMetric();
+ double prec = (double)this.truePos / (this.truePos + this.falsePos) * 100;
+ precision.setValue(prec);
+ precision.setType(PerformanceMetricType.PERCENT);
+ return precision;
+ }
+
+ private PerformanceMetric buildRecall() {
+ PerformanceMetric recall = new PerformanceMetric();
+ double rec = (double)this.truePos / (this.truePos + this.falseNeg) * 100;
+ recall.setValue(rec);
+ recall.setType(PerformanceMetricType.PERCENT);
+ return recall;
+ }
+
+ /***
+ * Builds the performance given a list of anomalies. When calculating entity anomalies, it counts the total number of anomalies
+ * only on the parent level but includes children anomalies when calculating the performance
+ * @param anomalies A list of anomalies
+ */
+ public Builder (List<MergedAnomalyResultDTO> anomalies) {
+ anomalies.stream()
+ .forEach(anomaly -> {
+ if (!anomaly.isChild()) {
+ this.numAnomalies++;
+ }
+
+ if (anomaly.getAnomalyResultSource() != null) {
+ if (AnomalyResultSource.USER_LABELED_ANOMALY.equals(anomaly.getAnomalyResultSource())) {
+ if (anomaly.getFeedback() == null || anomaly.getFeedback().getFeedbackType().isAnomaly()) {
+ // NOTE: includes user-created anomaly without feedback as false negative by default
+ this.falseNeg++;
+ this.respondedAnomalies++;
+ return;
+ }
Review comment:
There can be an edge case that the user labeled a reported anomaly as "NOT_ANOMALY", in that case, we still want to make `respondedAnomalies++`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] bryantachen closed pull request #6268: [TE] rest-api enhance implementation to calculate alert performance
Posted by GitBox <gi...@apache.org>.
bryantachen closed pull request #6268:
URL: https://github.com/apache/incubator-pinot/pull/6268
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org