You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/01/15 19:51:35 UTC
[GitHub] [incubator-pinot] jihaozh opened a new pull request #4988: [TE]
Support weekly anomaly detection
jihaozh opened a new pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988
This PR adds the capability to run anomaly detection on a weekly aggregated data on an hourly/daily metric. It will aggregate the data into weekly data and run the detection algorithm (Percentage change). Users can specify the desired week start day for aggregation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367156794
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
Review comment:
Good catch. Maybe we need to support AVG too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull
request #4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
vincentchenjl commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367160357
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
+ startMillis = window.getStart().withDayOfWeek(weekStart.getValue()).minusWeeks(1).getMillis();
+ }
+
MetricEntity me = MetricEntity.fromURN(metricUrn);
- MetricSlice slice = MetricSlice.from(me.getId(), window.getStartMillis(), window.getEndMillis(), me.getFilters(), timeGranularity);
+ MetricSlice slice = MetricSlice.from(me.getId(), startMillis, window.getEndMillis(), me.getFilters(), timeGranularity);
List<MetricSlice> slices = new ArrayList<>(this.baseline.scatter(slice));
slices.add(slice);
InputData data = this.dataFetcher.fetchData(new InputDataSpec().withTimeseriesSlices(slices)
.withMetricIdsForDataset(Collections.singletonList(slice.getMetricId())));
- DataFrame dfCurr = data.getTimeseries().get(slice).renameSeries(COL_VALUE, COL_CURR);
DataFrame dfBase = this.baseline.gather(slice, data.getTimeseries());
+ DataFrame dfCurr = data.getTimeseries().get(slice);
+ DatasetConfigDTO datasetConfig = data.getDatasetForMetricId().get(me.getId());
+ // aggregate data to specified weekly granularity
+ if (this.monitoringGranularity.endsWith("WEEKS")) {
Review comment:
Do we also support MONTHS?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh merged pull request #4988: [TE] Support
weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh merged pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367109943
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
+ startMillis = window.getStart().withDayOfWeek(weekStart.getValue()).minusWeeks(1).getMillis();
Review comment:
The startMills calculation should be done when creating the task. Why calculating this again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull
request #4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
vincentchenjl commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367136187
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DefaultDataProvider.java
##########
@@ -26,6 +26,7 @@
import com.google.common.collect.ArrayListMultimap;
import com.google.common.collect.Lists;
import com.google.common.collect.Multimap;
+import com.ibm.icu.util.TimeZone;
Review comment:
Are we using this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull
request #4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
vincentchenjl commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367158878
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
Review comment:
Can we pass the aggregation function as an argument
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367154505
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
Review comment:
They should be equivalent in functionality. This is more of java 8 style.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367589303
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
Review comment:
Sure. Changed to that and also support AVG
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367108045
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
+ }
+
+ /**
+ * Check if the aggregation result is complete or not, if not, remove it from the aggregated result.
+ *
+ * For example, say the weekStart is Monday and current data is available through Jan 8, Wednesday.
+ * the latest data time stamp will be Jan 8. The latest aggregation start time stamp should be Jan 6, Monday.
+ * In such case, the latest data point is incomplete and should be filtered. If the latest data time stamp is
+ * Jan 12, Sunday instead, the data is complete and good to use because the week's data is complete.
+ *
+ * @param df the aggregated data frame to check
+ * @param latestDataTimeStamp the latest data time stamp
+ * @param bucketTimeGranularity the metric's original granularity
+ * @param aggregationGranularityPeriod the granularity after aggregation
+ * @return the filtered data frame
+ */
+ public static DataFrame checkIncompleteAggregation(DataFrame df, long latestDataTimeStamp,
Review comment:
Consider renaming the function - it is not clear what is "check".
This is actually filterIncompleteAggregation?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367153559
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
+ }
+
+ /**
+ * Check if the aggregation result is complete or not, if not, remove it from the aggregated result.
+ *
+ * For example, say the weekStart is Monday and current data is available through Jan 8, Wednesday.
+ * the latest data time stamp will be Jan 8. The latest aggregation start time stamp should be Jan 6, Monday.
+ * In such case, the latest data point is incomplete and should be filtered. If the latest data time stamp is
+ * Jan 12, Sunday instead, the data is complete and good to use because the week's data is complete.
+ *
+ * @param df the aggregated data frame to check
+ * @param latestDataTimeStamp the latest data time stamp
+ * @param bucketTimeGranularity the metric's original granularity
+ * @param aggregationGranularityPeriod the granularity after aggregation
+ * @return the filtered data frame
+ */
+ public static DataFrame checkIncompleteAggregation(DataFrame df, long latestDataTimeStamp,
Review comment:
Yes. I'll rename it to `filterIncompleteAggregation `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367110394
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
Review comment:
Should we check the aggregation function? Are we assuming the aggregation is always SUM?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull
request #4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
vincentchenjl commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367622706
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/HoltWintersDetector.java
##########
@@ -114,31 +120,27 @@ public void init(HoltWintersDetectorSpec spec, InputDataFetcher dataFetcher) {
this.sensitivity = spec.getSensitivity();
this.monitoringGranularity = spec.getMonitoringGranularity();
- if (this.monitoringGranularity.equals("1_MONTHS")) {
+ if (this.monitoringGranularity.endsWith("MONTHS") || this.monitoringGranularity.endsWith("WEEKS")) {
Review comment:
Should we define `"MONTHS"` and `"WEEKS"` as constants?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367589303
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionUtils.java
##########
@@ -288,4 +288,39 @@ public static Period periodFromTimeUnit(int size, TimeUnit unit) {
return new Period(TimeUnit.MILLISECONDS.convert(size, unit));
}
}
+
+ /**
+ * Aggregate the time series data frame's value to specified granularity
+ * @param df the data frame
+ * @param origin the aggregation origin time stamp
+ * @param granularityPeriod the aggregation granularity in period
+ * @return the aggregated time series data frame
+ */
+ public static DataFrame aggregateByPeriod(DataFrame df, long origin, Period granularityPeriod) {
+ return df.groupByPeriod(df.getLongs(COL_TIME), new DateTime(origin), granularityPeriod).sum(COL_TIME, COL_VALUE);
Review comment:
Sure. Made the change to support other aggregation functions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367154967
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
+ startMillis = window.getStart().withDayOfWeek(weekStart.getValue()).minusWeeks(1).getMillis();
Review comment:
Yes. This is to ensure that the window is aligned to the specified boundary because in preview it can be adjusted to any day.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367589594
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DefaultDataProvider.java
##########
@@ -26,6 +26,7 @@
import com.google.common.collect.ArrayListMultimap;
import com.google.common.collect.Lists;
import com.google.common.collect.Multimap;
+import com.ibm.icu.util.TimeZone;
Review comment:
Good catch. Removed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull
request #4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
vincentchenjl commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367619494
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
+ startMillis = window.getStart().withDayOfWeek(weekStart.getValue()).minusWeeks(1).getMillis();
+ }
+
MetricEntity me = MetricEntity.fromURN(metricUrn);
- MetricSlice slice = MetricSlice.from(me.getId(), window.getStartMillis(), window.getEndMillis(), me.getFilters(), timeGranularity);
+ MetricSlice slice = MetricSlice.from(me.getId(), startMillis, window.getEndMillis(), me.getFilters(), timeGranularity);
List<MetricSlice> slices = new ArrayList<>(this.baseline.scatter(slice));
slices.add(slice);
InputData data = this.dataFetcher.fetchData(new InputDataSpec().withTimeseriesSlices(slices)
.withMetricIdsForDataset(Collections.singletonList(slice.getMetricId())));
- DataFrame dfCurr = data.getTimeseries().get(slice).renameSeries(COL_VALUE, COL_CURR);
DataFrame dfBase = this.baseline.gather(slice, data.getTimeseries());
+ DataFrame dfCurr = data.getTimeseries().get(slice);
+ DatasetConfigDTO datasetConfig = data.getDatasetForMetricId().get(me.getId());
+ // aggregate data to specified weekly granularity
+ if (this.monitoringGranularity.endsWith("WEEKS")) {
Review comment:
Sure. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367702491
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/HoltWintersDetector.java
##########
@@ -114,31 +120,27 @@ public void init(HoltWintersDetectorSpec spec, InputDataFetcher dataFetcher) {
this.sensitivity = spec.getSensitivity();
this.monitoringGranularity = spec.getMonitoringGranularity();
- if (this.monitoringGranularity.equals("1_MONTHS")) {
+ if (this.monitoringGranularity.endsWith("MONTHS") || this.monitoringGranularity.endsWith("WEEKS")) {
Review comment:
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
xiaohui-sun commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367108271
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
Review comment:
Why use Objects.notNull instead of this.weekStart == null?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request
#4988: [TE] Support weekly anomaly detection
Posted by GitBox <gi...@apache.org>.
jihaozh commented on a change in pull request #4988: [TE] Support weekly anomaly detection
URL: https://github.com/apache/incubator-pinot/pull/4988#discussion_r367591035
##########
File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/PercentageChangeRuleDetector.java
##########
@@ -76,16 +80,34 @@
@Override
public DetectionResult runDetection(Interval window, String metricUrn) {
+ long startMillis = window.getStartMillis();
+
+ // align start day to the user specified week start
+ if (Objects.nonNull(this.weekStart)) {
+ startMillis = window.getStart().withDayOfWeek(weekStart.getValue()).minusWeeks(1).getMillis();
+ }
+
MetricEntity me = MetricEntity.fromURN(metricUrn);
- MetricSlice slice = MetricSlice.from(me.getId(), window.getStartMillis(), window.getEndMillis(), me.getFilters(), timeGranularity);
+ MetricSlice slice = MetricSlice.from(me.getId(), startMillis, window.getEndMillis(), me.getFilters(), timeGranularity);
List<MetricSlice> slices = new ArrayList<>(this.baseline.scatter(slice));
slices.add(slice);
InputData data = this.dataFetcher.fetchData(new InputDataSpec().withTimeseriesSlices(slices)
.withMetricIdsForDataset(Collections.singletonList(slice.getMetricId())));
- DataFrame dfCurr = data.getTimeseries().get(slice).renameSeries(COL_VALUE, COL_CURR);
DataFrame dfBase = this.baseline.gather(slice, data.getTimeseries());
+ DataFrame dfCurr = data.getTimeseries().get(slice);
+ DatasetConfigDTO datasetConfig = data.getDatasetForMetricId().get(me.getId());
+ // aggregate data to specified weekly granularity
+ if (this.monitoringGranularity.endsWith("WEEKS")) {
Review comment:
MONTHS is supported only if the metric itself is a monthly metric. The aggregation to monthly is not supported. I think that's out of the scope of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org