You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ji...@apache.org on 2020/09/02 17:24:10 UTC
[incubator-pinot] branch severity-alerter updated (fe3df96 ->
5511721)
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a change to branch severity-alerter
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
discard fe3df96 refactor
discard 45d34f0 severity alerter & unit tests
discard 6ef83ea add anomaly notification table
add addf67f Simplify compression logic for anomaly timelines view. Now we only keep the values for the selected timestamps after compression, instead of taking an average that could cause value shifts (#5906)
add e7e9775 Refactored code for overwritten configs (#5875)
add 331b874 Expose ResultSetStats in the Pinot client's ResultSetGroup (#5892)
add d733a24 [TE] anomalies page - fix feedback is not populated if retrived by the method findByIds (#5903)
add ec27472 [TE] frontend - harleyjj/units - Add notes and tooltips explaining SI prefix symbols (#5886)
add a9818f5 [TE] add anomaly detection as a service - endpoints for managing online detection data (#5894)
add 262678b Adds note about using intellij with testng (#5912)
add a8974c7 [TE] Updated ThirdEye docker launch script to accept MODE as a second arg (#5914)
add 4c4777d [TE] frontend - harleyjj/alert-details - fix bug in forecast dates (#5916)
add 1b3fb99 [TE] Remove deprecated classes under /detection/algorithm (#5908)
add afc7539 [TE] Fix corner case in SLA alert - round up datasetLastRefreshTime (#5909)
add c425b0f [TE] add anomaly detection as a service - new table for storing ad-hoc data (#5866)
add 9e65bdf [TE] cube api bug fix to fetch by dataset name (#5921)
add ba55d58 Updating docusaurus and adding pinot navbar logos (#5918)
add dafbef1 Add 0.5 license and notice. (#5905)
add ced3a70 Fix CSV and JSON converter on BYTES column (#5931)
add 1b38909 [TE] ThirdEye User Config (Yaml) Validation (#5930)
add 9cd46b1 [TE] Remove incessant DEBUG level logging (#5932)
add 21a372b Starts Broker and Server in parallel when using ServiceManager (#5917)
add e71622d support for local.directory.sequence.id (#5927)
add 8f0145e [TE] update schema to support window parameters (#5933)
add e4ea9fc [TE] Multiple changes to get TE running in a docker container (#5924)
add 6b78dcc Add max qps bucket count (#5922)
add d47e751 Add an explicit license check exclusion for github related templates. (#5936)
add 701ffcb Revert the commit 41dc432e775aeeb841f57bd25525832950698739 (#5938)
add 347eff1 Fixing the issue that transform UDFs are parsed as function name 'OTHER', not the real function names (#5940)
add c3446ba [TE] polish the rca template in email (#5939)
add a892fb4 Add Range Indexing support for raw values (#5853)
add d2ed761 adding autocomplete in sql editor (#5810)
add 2cbf0b5 5804: Prints file name of invalid json file when parsing (#5910)
add 86a01ff Add IdSet and IdSetAggregationFunction (#5926)
add a31c06a Bump up swagger-ui version and update related code (#5941)
add 42c6424 Make realtime threshold property names less ambiguous (#5953)
add 032a7bf Move to natural POM versions (#5945)
add d444285 Incorporating embedded exception while trying to fetch stream offset (#5956)
new 8bc3d48 add anomaly notification table
new f614acb severity alerter & unit tests
new 5511721 refactor
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fe3df96)
\
N -- N -- N refs/heads/severity-alerter (5511721)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.travis/.travis_nightly_build.sh | 2 +-
LICENSE-binary | 171 +----
NOTICE-binary | 97 ++-
docker/images/pinot-thirdeye/README.md | 2 +-
docker/images/pinot-thirdeye/bin/start-thirdeye.sh | 73 +-
docs/dev_env.rst | 3 +
licenses-binary/LICENSE-gpl-2.0.txt | 125 ----
pinot-broker/pom.xml | 2 +-
.../HelixExternalViewBasedQueryQuotaManager.java | 15 +-
.../apache/pinot/broker/queryquota/HitCounter.java | 26 +-
.../pinot/broker/queryquota/MaxHitRateTracker.java | 81 +++
.../pinot/broker/queryquota/QueryQuotaEntity.java | 18 +-
.../broker/queryquota/MaxHitRateTrackerTest.java | 59 ++
pinot-clients/pinot-java-client/pom.xml | 2 +-
.../org/apache/pinot/client/BrokerResponse.java | 6 +
.../java/org/apache/pinot/client/Connection.java | 2 +-
.../org/apache/pinot/client/ExecutionStats.java | 142 ++++
.../org/apache/pinot/client/ResultSetGroup.java | 8 +
.../apache/pinot/client/PreparedStatementTest.java | 4 +-
.../apache/pinot/client/ResultSetGroupTest.java | 12 +-
pinot-clients/pinot-jdbc-client/pom.xml | 2 +-
pinot-clients/pom.xml | 2 +-
pinot-common/pom.xml | 2 +-
.../common/function/AggregationFunctionType.java | 1 +
.../pinot/common/metrics/AbstractMetrics.java | 25 +
.../apache/pinot/common/metrics/BrokerGauge.java | 1 +
.../apache/pinot/sql/parsers/CalciteSqlParser.java | 1 +
.../src/main/resources/swagger-ui/index.html | 149 ++--
.../pinot/sql/parsers/CalciteSqlCompilerTest.java | 13 +
pinot-connectors/pinot-spark-connector/pom.xml | 2 +-
pinot-connectors/pom.xml | 2 +-
pinot-controller/pom.xml | 2 +-
.../api/ControllerAdminApiApplication.java | 7 +
.../realtime/PinotLLCRealtimeSegmentManager.java | 2 +-
.../SegmentSizeBasedFlushThresholdUpdater.java | 2 +-
.../controller/recommender/io/InputManager.java | 68 +-
.../recommender/rules/RulesToExecute.java | 2 +-
.../rules/impl/InvertedSortedIndexJointRule.java | 80 ++-
.../recommender/rules/impl/KafkaPartitionRule.java | 2 +-
.../NoDictionaryOnHeapDictionaryJointRule.java | 13 +-
.../rules/impl/PinotTablePartitionRule.java | 37 +-
.../recommender/rules/io/configs/IndexConfig.java | 14 +-
.../rules/io/configs/PartitionConfig.java | 19 +
.../params/InvertedSortedIndexJointRuleParams.java | 6 +-
.../rules/io/params/PartitionRuleParams.java | 2 +-
.../rules/io/params/RecommenderConstants.java | 12 +-
.../utils/QueryInvertedSortedIndexRecommender.java | 43 +-
pinot-controller/src/main/resources/api/index.html | 148 ++--
.../src/main/resources/app/pages/Query.tsx | 161 +++--
.../src/main/resources/app/styles/styles.css | 61 ++
.../main/resources/app/utils/PinotMethodUtils.ts | 11 +-
.../src/main/resources/app/utils/Utils.tsx | 163 ++++-
pinot-controller/src/main/resources/package.json | 1 +
.../segment/FlushThresholdUpdaterTest.java | 9 +-
.../controller/recommender/TestConfigEngine.java | 11 +-
.../recommenderInput/BloomFilterInput.json | 2 +-
.../recommenderInput/DataSizeCalculationInput.json | 2 +-
...titionRuleInput.json => EmptyQueriesInput.json} | 69 +-
.../recommenderInput/KafkaPartitionRuleInput.json | 2 +-
.../recommenderInput/KafkaPartitionRuleInput2.json | 2 +-
...NoDictionaryOnHeapDictionaryJointRuleInput.json | 2 +-
.../recommenderInput/SortedInvertedIndexInput.json | 2 +-
pinot-core/pom.xml | 2 +-
.../apache/pinot/core/common/ObjectSerDeUtils.java | 39 +-
.../filter/RangeIndexBasedFilterOperator.java | 52 +-
.../predicate/RangePredicateEvaluatorFactory.java | 40 +-
.../function/AggregationFunctionFactory.java | 2 +
.../function/IdSetAggregationFunction.java | 352 ++++++++++
.../BrokerRequestToQueryContextConverter.java | 9 +-
.../core/query/utils/idset/BloomFilterIdSet.java | 235 +++++++
.../pinot/core/query/utils/idset/EmptyIdSet.java | 85 +++
.../apache/pinot/core/query/utils/idset/IdSet.java | 153 +++++
.../pinot/core/query/utils/idset/IdSets.java | 236 +++++++
.../utils/idset/Roaring64NavigableMapIdSet.java | 114 +++
.../core/query/utils/idset/RoaringBitmapIdSet.java | 101 +++
...va => DictionaryBasedInvertedIndexCreator.java} | 24 +-
.../core/segment/creator/InvertedIndexCreator.java | 52 +-
.../creator/RawValueBasedInvertedIndexCreator.java | 62 ++
.../creator/impl/SegmentColumnarIndexCreator.java | 21 +-
.../inv/OffHeapBitmapInvertedIndexCreator.java | 8 +-
.../impl/inv/OnHeapBitmapInvertedIndexCreator.java | 6 +-
.../creator/impl/inv/RangeIndexCreator.java | 153 +++--
.../impl/inv/text/LuceneTextIndexCreator.java | 35 +-
.../loader/invertedindex/RangeIndexHandler.java | 129 +++-
.../pinot/core/query/utils/idset/IdSetTest.java | 202 ++++++
.../core/realtime/stream/StreamConfigTest.java | 49 +-
.../index/creator/RangeIndexCreatorTest.java | 402 ++++++++---
.../org/apache/pinot/queries/IdSetQueriesTest.java | 401 +++++++++++
pinot-distribution/pom.xml | 2 +-
pinot-integration-tests/pom.xml | 2 +-
.../src/test/resources/index.html | 148 ++--
pinot-minion/pom.xml | 2 +-
pinot-perf/pom.xml | 2 +-
.../pinot-batch-ingestion-common/pom.xml | 2 +-
.../batch/common/SegmentGenerationTaskRunner.java | 3 +
.../pinot-batch-ingestion-hadoop/pom.xml | 2 +-
.../pinot-batch-ingestion-spark/pom.xml | 2 +-
.../spark/SparkSegmentGenerationJobRunner.java | 31 +-
.../pinot-batch-ingestion-standalone/pom.xml | 2 +-
pinot-plugins/pinot-batch-ingestion/pom.xml | 2 +-
.../v0_deprecated/pinot-hadoop/pom.xml | 2 +-
.../v0_deprecated/pinot-ingestion-common/pom.xml | 2 +-
.../v0_deprecated/pinot-spark/pom.xml | 2 +-
.../pinot-batch-ingestion/v0_deprecated/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 2 +-
pinot-plugins/pinot-file-system/pom.xml | 2 +-
.../pinot-input-format/pinot-avro-base/pom.xml | 2 +-
.../pinot-input-format/pinot-avro/pom.xml | 2 +-
.../pinot-confluent-avro/pom.xml | 2 +-
pinot-plugins/pinot-input-format/pinot-csv/pom.xml | 2 +-
.../pinot-input-format/pinot-json/pom.xml | 2 +-
pinot-plugins/pinot-input-format/pinot-orc/pom.xml | 2 +-
.../pinot-input-format/pinot-parquet/pom.xml | 2 +-
.../pinot-input-format/pinot-protobuf/pom.xml | 2 +-
.../pinot-input-format/pinot-thrift/pom.xml | 2 +-
pinot-plugins/pinot-input-format/pom.xml | 2 +-
.../pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 2 +-
.../pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 2 +-
.../pinot-kafka-base/pom.xml | 2 +-
pinot-plugins/pinot-stream-ingestion/pom.xml | 2 +-
pinot-plugins/pom.xml | 2 +-
pinot-server/pom.xml | 2 +-
.../server/starter/helix/AdminApiApplication.java | 7 +
pinot-server/src/main/resources/api/index.html | 148 ++--
pinot-spi/pom.xml | 2 +-
.../spi/stream/PartitionLevelStreamConfig.java | 2 +-
.../org/apache/pinot/spi/stream/StreamConfig.java | 68 +-
.../pinot/spi/stream/StreamConfigProperties.java | 13 +-
pinot-tools/pom.xml | 2 +-
.../tools/admin/command/CreateSegmentCommand.java | 2 +-
.../admin/command/StartServiceManagerCommand.java | 132 +++-
.../converter/PinotSegmentToAvroConverter.java | 13 +-
.../converter/PinotSegmentToCsvConverter.java | 47 +-
.../converter/PinotSegmentToJsonConverter.java | 16 +-
.../pinot/tools/TestCreateSegmentCommand.java | 64 ++
.../converter/PinotSegmentConverterTest.java | 195 ++++++
.../resources/test_data/test_invalid_data.json | 2 +
pom.xml | 9 +-
thirdeye/pom.xml | 1 +
.../app/pods/components/alert-details/component.js | 6 +-
.../app/pods/components/alert-details/template.hbs | 3 +
.../pods/components/anomaly-summary/template.hbs | 4 +-
.../pods/components/rootcause-anomaly/template.hbs | 4 +
.../anomalies-table/current-wow/template.hbs | 5 +-
.../app/pods/partials/si-note/template.hbs | 1 +
.../app/pods/partials/si-tooltip/template.hbs | 8 +
.../app/styles/components/rootcause-header.scss | 1 +
thirdeye/thirdeye-pinot/pom.xml | 21 +-
.../anomaly/ThirdEyeAnomalyApplication.java | 18 +-
.../anomaly/ThirdEyeAnomalyConfiguration.java | 45 --
.../trigger/DataAvailabilityTaskScheduler.java | 3 +
.../anomaly/monitor/MonitorTaskRunner.java | 10 +
.../views/CondensedAnomalyTimelinesView.java | 8 +-
.../api/detection/AnomalyDetectionResource.java | 584 ++++++++++++----
.../thirdeye/common/ThirdEyeConfiguration.java | 45 +-
.../dashboard/ThirdEyeDashboardConfiguration.java | 1 -
.../dashboard/resources/SummaryResource.java | 2 +-
.../dashboard/resources/v2/AnomaliesResource.java | 1 -
.../dashboard/resources/v2/RootCauseResource.java | 2 +-
.../pinot/thirdeye/datalayer/ScriptRunner.java | 50 +-
.../datalayer/bao/MergedAnomalyResultManager.java | 2 -
.../datalayer/bao/OnlineDetectionDataManager.java | 9 +
.../bao/jdbc/MergedAnomalyResultManagerImpl.java | 3 +-
.../bao/jdbc/OnlineDetectionDataManagerImpl.java | 25 +
.../thirdeye/datalayer/dao/GenericPojoDao.java | 16 +-
.../datalayer/dto/OnlineDetectionDataDTO.java | 6 +
.../datalayer/entity/OnlineDetectionDataIndex.java | 22 +
.../thirdeye/datalayer/pojo/MetricConfigBean.java | 10 -
.../datalayer/pojo/OnlineDetectionDataBean.java | 51 ++
.../thirdeye/datalayer/util/DaoProviderUtil.java | 7 +-
.../datalayer/util/EntityMappingHolder.java | 2 -
.../thirdeye/datalayer/util/SqlQueryBuilder.java | 3 -
.../pinot/thirdeye/datasource/DAORegistry.java | 6 +
.../online/OnlineThirdEyeDataSource.java | 35 +-
.../thirdeye/detection/DefaultDataProvider.java | 3 +-
.../detection/DetectionPipelineLoader.java | 2 +-
.../pinot/thirdeye/detection/DetectionUtils.java | 4 +-
.../detection/algorithm/BaselineAlgorithm.java | 139 ----
.../algorithm/BaselineRuleFilterWrapper.java | 117 ----
.../detection/algorithm/MovingWindowAlgorithm.java | 762 ---------------------
.../algorithm/RuleBasedFilterWrapper.java | 91 ---
.../detection/algorithm/ThresholdAlgorithm.java | 93 ---
.../algorithm/ThresholdRuleFilterWrapper.java | 73 --
.../registry/DetectionAlertRegistry.java | 1 -
.../components/AbsoluteChangeRuleDetector.java | 2 +-
.../detection/components/HoltWintersDetector.java | 2 +-
.../components/MeanVarianceRuleDetector.java | 1 -
.../components/PercentageChangeRuleDetector.java | 2 +-
.../components/ThresholdRuleDetector.java | 2 +-
.../components/DataSlaQualityChecker.java | 40 +-
.../spec/DataSlaQualityCheckerSpec.java | 7 +-
.../validators/ConfigValidationException.java | 57 ++
.../validators/ConfigValidationUtils.java | 57 ++
.../detection/validators/ConfigValidator.java | 25 +-
.../validators/DetectionConfigValidator.java | 226 ++----
.../validators/SubscriptionConfigValidator.java | 117 ++--
.../validators/ThirdEyeUserConfigValidator.java | 75 ++
.../detection/wrapper/AnomalyDetectorWrapper.java | 5 +-
.../thirdeye/detection/yaml/YamlResource.java | 286 ++++----
.../yaml/translator/ConfigTranslator.java | 26 +-
.../yaml/translator/DetectionConfigTranslator.java | 17 +-
.../translator/DetectionMetricAttributeHolder.java | 2 +-
.../translator/SubscriptionConfigTranslator.java | 23 +-
.../builder/DetectionConfigPropertiesBuilder.java | 4 +-
.../detector/metric-anomalies-template.ftl | 72 +-
.../src/main/resources/schema/create-schema.sql | 14 +-
.../detection/detection-config-schema.json | 255 +++++++
.../subscription/subscription-config-schema.json | 113 +++
.../views/TestCondensedAnomalyTimelinesView.java | 16 +-
.../detection/AnomalyDetectionResourceTest.java | 270 +++++---
.../pinot/thirdeye/datalayer/DaoTestUtils.java | 14 +-
.../pinot/thirdeye/datalayer/bao/DAOTestBase.java | 6 +-
.../bao/TestOnlineDetectionDataManager.java | 63 ++
.../pinot/thirdeye/detection/DataProviderTest.java | 2 -
.../pinot/thirdeye/detection/MockDataProvider.java | 11 +-
.../detection/algorithm/BaselineAlgorithmTest.java | 139 ----
.../algorithm/MovingWindowAlgorithmTest.java | 414 -----------
.../algorithm/ThresholdAlgorithmTest.java | 89 ---
.../dataquality/DataQualityTaskRunnerTest.java | 88 +--
.../MergeDimensionThresholdIntegrationTest.java | 158 -----
.../validators/DetectionConfigValidatorTest.java | 172 +++--
.../SubscriptionConfigValidatorTest.java | 272 ++++++++
.../thirdeye/detection/yaml/YamlResourceTest.java | 52 +-
.../translator/DetectionConfigTranslatorTest.java | 5 +-
.../YamlDetectionAlertConfigTranslatorTest.java | 7 +-
.../integration/NotificationTaskSchedulerTest.java | 12 +-
.../thirdeye/api/detection/payload-bad-custom.json | 6 +-
.../pinot/thirdeye/api/detection/payload-bad.json | 6 +-
.../api/detection/payload-good-custom.json | 6 +-
.../pinot/thirdeye/api/detection/payload-good.json | 6 +-
.../api/detection/payload-update-test1.json | 2 +
.../api/detection/payload-update-test2.json | 2 +
.../api/detection/payload-update-test3.json | 3 +
.../mergeDimensionThresholdProperties.json | 31 -
.../detection/migrated-detection-config-1.json | 2 +-
.../detection/migrated-detection-config-2.json | 2 +-
....yaml => composite-detection-config-bad-1.yaml} | 8 +-
...yaml => composite-detection-config-good-1.yaml} | 8 +-
.../validators/detection-config-bad-1.yaml | 61 ++
.../validators/detection-config-good-1.yaml | 82 +++
.../validators/entity-pipeline-config-2.yaml | 4 +-
.../yaml/subscription/subscription-config-1.yaml | 12 +-
.../yaml/subscription/subscription-config-2.yaml | 8 +-
.../yaml/subscription/subscription-config-3.yaml | 8 +-
.../yaml/subscription/subscription-config-4.yaml | 8 +-
.../yaml/subscription/subscription-config-5.yaml | 8 +-
.../compositePipelineTranslatorTestResult-5.json | 24 +-
.../yaml/translator/pipeline-config-1.yaml | 5 +-
.../yaml/translator/pipeline-config-2.yaml | 2 +-
.../yaml/translator/pipeline-config-3.yaml | 2 +-
.../yaml/translator/pipeline-config-5.yaml | 4 +-
.../detection/yaml/translator/sla-config-5.yaml | 8 +-
.../yaml/translator/sla-config-translated-5.json | 12 +-
.../src/test/resources/sample-alert-config.yml | 2 +-
...rca-highlights-cube-algo-response-rendered.html | 84 +--
...st-email-rca-highlights-cube-algo-response.json | 2 +-
website/docusaurus.config.js | 23 +-
website/package.json | 12 +-
website/static/img/pinot-navbar-logo-722f37.png | Bin 0 -> 42885 bytes
website/static/img/pinot-navbar-logo-722f37.svg | 21 +
website/static/img/pinot-overview.png | Bin 0 -> 128920 bytes
264 files changed, 7250 insertions(+), 4692 deletions(-)
delete mode 100644 licenses-binary/LICENSE-gpl-2.0.txt
create mode 100644 pinot-broker/src/main/java/org/apache/pinot/broker/queryquota/MaxHitRateTracker.java
create mode 100644 pinot-broker/src/test/java/org/apache/pinot/broker/queryquota/MaxHitRateTrackerTest.java
create mode 100644 pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ExecutionStats.java
copy pinot-controller/src/test/resources/recommenderInput/{KafkaPartitionRuleInput.json => EmptyQueriesInput.json} (53%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/IdSetAggregationFunction.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/utils/idset/BloomFilterIdSet.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/utils/idset/EmptyIdSet.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/utils/idset/IdSet.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/utils/idset/IdSets.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/utils/idset/Roaring64NavigableMapIdSet.java
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/query/utils/idset/RoaringBitmapIdSet.java
copy pinot-core/src/main/java/org/apache/pinot/core/segment/creator/{InvertedIndexCreator.java => DictionaryBasedInvertedIndexCreator.java} (82%)
create mode 100644 pinot-core/src/main/java/org/apache/pinot/core/segment/creator/RawValueBasedInvertedIndexCreator.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/core/query/utils/idset/IdSetTest.java
create mode 100644 pinot-core/src/test/java/org/apache/pinot/queries/IdSetQueriesTest.java
create mode 100644 pinot-tools/src/test/java/org/apache/pinot/tools/TestCreateSegmentCommand.java
create mode 100644 pinot-tools/src/test/java/org/apache/pinot/tools/segment/converter/PinotSegmentConverterTest.java
create mode 100644 pinot-tools/src/test/resources/test_data/test_invalid_data.json
create mode 100644 thirdeye/thirdeye-frontend/app/pods/partials/si-note/template.hbs
create mode 100644 thirdeye/thirdeye-frontend/app/pods/partials/si-tooltip/template.hbs
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/OnlineDetectionDataManager.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/jdbc/OnlineDetectionDataManagerImpl.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/OnlineDetectionDataDTO.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/entity/OnlineDetectionDataIndex.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/OnlineDetectionDataBean.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/BaselineAlgorithm.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/BaselineRuleFilterWrapper.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/MovingWindowAlgorithm.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/RuleBasedFilterWrapper.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/ThresholdAlgorithm.java
delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/algorithm/ThresholdRuleFilterWrapper.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/validators/ConfigValidationException.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/validators/ConfigValidationUtils.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/validators/ThirdEyeUserConfigValidator.java
create mode 100644 thirdeye/thirdeye-pinot/src/main/resources/validators/detection/detection-config-schema.json
create mode 100644 thirdeye/thirdeye-pinot/src/main/resources/validators/subscription/subscription-config-schema.json
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/datalayer/bao/TestOnlineDetectionDataManager.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/BaselineAlgorithmTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/MovingWindowAlgorithmTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/algorithm/ThresholdAlgorithmTest.java
delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/integration/MergeDimensionThresholdIntegrationTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/validators/SubscriptionConfigValidatorTest.java
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-update-test1.json
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-update-test2.json
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-update-test3.json
delete mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/integration/mergeDimensionThresholdProperties.json
copy thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/validators/{entity-pipeline-config-2.yaml => composite-detection-config-bad-1.yaml} (93%)
copy thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/validators/{entity-pipeline-config-2.yaml => composite-detection-config-good-1.yaml} (88%)
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/validators/detection-config-bad-1.yaml
create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/detection/validators/detection-config-good-1.yaml
create mode 100644 website/static/img/pinot-navbar-logo-722f37.png
create mode 100644 website/static/img/pinot-navbar-logo-722f37.svg
create mode 100644 website/static/img/pinot-overview.png
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 02/03: severity alerter & unit tests
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch severity-alerter
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit f614acb370c6865b3d5444a82fa42ff02232f573
Author: Jihao Zhang <ji...@linkedin.com>
AuthorDate: Tue Sep 1 13:54:04 2020 -0700
severity alerter & unit tests
---
.../anomaly/monitor/MonitorTaskRunner.java | 10 +
.../pinot/thirdeye/constant/AnomalySeverity.java | 35 ++++
.../datalayer/dto/MergedAnomalyResultDTO.java | 10 +
.../datalayer/pojo/MergedAnomalyResultBean.java | 10 +
.../detection/DetectionPipelineTaskRunner.java | 30 +++
.../thirdeye/detection/DetectionResource.java | 22 +-
.../detection/alert/DetectionAlertJob.java | 14 +-
.../alert/filter/AnomalySeverityAlertFilter.java | 163 +++++++++++++++
.../translator/SubscriptionConfigTranslator.java | 3 +-
.../detection/alert/filter/AlertFilterUtils.java | 9 +-
.../filter/AnomalySeverityAlertFilterTest.java | 232 +++++++++++++++++++++
11 files changed, 526 insertions(+), 12 deletions(-)
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
index a5c1ca3..7e47f3b 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/monitor/MonitorTaskRunner.java
@@ -291,6 +291,16 @@ public class MonitorTaskRunner implements TaskRunner {
} catch (Exception e) {
LOG.error("Exception when deleting old online detection data.", e);
}
+
+ // Delete old anomaly subscription notifications.
+ try {
+ int deletedRecords = DAO_REGISTRY.getAnomalySubscriptionGroupNotificationManager()
+ .deleteRecordsOlderThanDays(monitorTaskInfo.getDefaultRetentionDays());
+ LOG.info("Deleted {} anomaly subscription notifications that are older than {} days.", deletedRecords,
+ monitorTaskInfo.getDefaultRetentionDays());
+ } catch (Exception e) {
+ LOG.error("Exception when deleting old anomaly subscription notifications.", e);
+ }
}
private Map<Long, JobDTO> findScheduledJobsWithinDays(int days) {
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/constant/AnomalySeverity.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/constant/AnomalySeverity.java
new file mode 100644
index 0000000..48e542e
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/constant/AnomalySeverity.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.constant;
+
+public enum AnomalySeverity {
+ LOW("low"), MODERATE("moderate"), HIGH("high"), CRITICAL("critical"), DEFAULT("default");
+
+ String userReadableName;
+
+ AnomalySeverity(String userReadableName) {
+ this.userReadableName = userReadableName;
+ }
+
+ public String getUserReadableName() {
+ return this.userReadableName;
+ }
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/MergedAnomalyResultDTO.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/MergedAnomalyResultDTO.java
index 415a252..620b0de 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/MergedAnomalyResultDTO.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/MergedAnomalyResultDTO.java
@@ -44,6 +44,8 @@ public class MergedAnomalyResultDTO extends MergedAnomalyResultBean implements A
private Set<MergedAnomalyResultDTO> children = new HashSet<>();
+ private boolean renotify;
+
public MergedAnomalyResultDTO() {
setCreatedTime(System.currentTimeMillis());
}
@@ -94,6 +96,14 @@ public class MergedAnomalyResultDTO extends MergedAnomalyResultBean implements A
return function;
}
+ public boolean shouldRenotify() {
+ return renotify;
+ }
+
+ public void setRenotify(boolean renotify) {
+ this.renotify = renotify;
+ }
+
@Deprecated
public void setFunction(AnomalyFunctionDTO function) {
this.function = function;
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/MergedAnomalyResultBean.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/MergedAnomalyResultBean.java
index 834356d..e29de67 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/MergedAnomalyResultBean.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/MergedAnomalyResultBean.java
@@ -29,6 +29,7 @@ import java.util.Map;
import java.util.Objects;
import java.util.Set;
import org.apache.commons.lang3.ObjectUtils;
+import org.apache.pinot.thirdeye.constant.AnomalySeverity;
public class MergedAnomalyResultBean extends AbstractBean implements Comparable<MergedAnomalyResultBean>, Serializable {
@@ -60,6 +61,7 @@ public class MergedAnomalyResultBean extends AbstractBean implements Comparable<
private Set<Long> childIds; // ids of the anomalies this anomaly merged from
private boolean isChild;
private AnomalyType type;
+ private AnomalySeverity severity = AnomalySeverity.DEFAULT;
public Set<Long> getChildIds() {
return childIds;
@@ -264,6 +266,14 @@ public class MergedAnomalyResultBean extends AbstractBean implements Comparable<
this.type = type;
}
+ public AnomalySeverity getSeverity() {
+ return severity;
+ }
+
+ public void setSeverity(AnomalySeverity severity) {
+ this.severity = severity;
+ }
+
@Override
public int hashCode() {
return Objects.hash(getId(), startTime, endTime, collection, metric, dimensions, score, impactToGlobal, avgBaselineVal, avgCurrentVal, anomalyResultSource, metricUrn, detectionConfigId, childIds, isChild);
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java
index 86126c7..183be6f 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java
@@ -27,6 +27,7 @@ import org.apache.pinot.thirdeye.anomaly.task.TaskInfo;
import org.apache.pinot.thirdeye.anomaly.task.TaskResult;
import org.apache.pinot.thirdeye.anomaly.task.TaskRunner;
import org.apache.pinot.thirdeye.anomaly.utils.ThirdeyeMetricsUtil;
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
import org.apache.pinot.thirdeye.datalayer.bao.DatasetConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.DetectionConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.EvaluationManager;
@@ -34,9 +35,11 @@ import org.apache.pinot.thirdeye.datalayer.bao.EventManager;
import org.apache.pinot.thirdeye.datalayer.bao.MergedAnomalyResultManager;
import org.apache.pinot.thirdeye.datalayer.bao.MetricConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.TaskManager;
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.EvaluationDTO;
import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
+import org.apache.pinot.thirdeye.datalayer.util.Predicate;
import org.apache.pinot.thirdeye.datasource.DAORegistry;
import org.apache.pinot.thirdeye.datasource.ThirdEyeCacheRegistry;
import org.apache.pinot.thirdeye.datasource.loader.AggregationLoader;
@@ -57,6 +60,7 @@ public class DetectionPipelineTaskRunner implements TaskRunner {
private final MergedAnomalyResultManager anomalyDAO;
private final EvaluationManager evaluationDAO;
private final TaskManager taskDAO;
+ private final AnomalySubscriptionGroupNotificationManager anomalySubscriptionGroupNotificationDAO;
private final DetectionPipelineLoader loader;
private final DataProvider provider;
private final ModelMaintenanceFlow maintenanceFlow;
@@ -76,6 +80,8 @@ public class DetectionPipelineTaskRunner implements TaskRunner {
this.anomalyDAO = DAORegistry.getInstance().getMergedAnomalyResultDAO();
this.evaluationDAO = DAORegistry.getInstance().getEvaluationManager();
this.taskDAO = DAORegistry.getInstance().getTaskDAO();
+ this.anomalySubscriptionGroupNotificationDAO =
+ DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
MetricConfigManager metricDAO = DAORegistry.getInstance().getMetricConfigDAO();
DatasetConfigManager datasetDAO = DAORegistry.getInstance().getDatasetConfigDAO();
EventManager eventDAO = DAORegistry.getInstance().getEventDAO();
@@ -112,6 +118,8 @@ public class DetectionPipelineTaskRunner implements TaskRunner {
this.loader = loader;
this.provider = provider;
this.taskDAO = DAORegistry.getInstance().getTaskDAO();
+ this.anomalySubscriptionGroupNotificationDAO =
+ DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
this.maintenanceFlow = new ModelRetuneFlow(this.provider, DetectionRegistry.getInstance());
}
@@ -167,6 +175,28 @@ public class DetectionPipelineTaskRunner implements TaskRunner {
}
this.detectionDAO.update(config);
+ // re-notify the anomalies if any
+ for (MergedAnomalyResultDTO anomaly : result.getAnomalies()) {
+ // if an anomaly should be re-notified, update the notification lookup table in the database
+ if (anomaly.shouldRenotify()) {
+ List<AnomalySubscriptionGroupNotificationDTO> subscriptionGroupNotificationDTOs =
+ this.anomalySubscriptionGroupNotificationDAO.findByPredicate(Predicate.EQ("anomalyId", anomaly.getId()));
+ AnomalySubscriptionGroupNotificationDTO anomalyNotificationDTO;
+ if (subscriptionGroupNotificationDTOs.isEmpty()) {
+ // create a new record if it is not existed yet.
+ anomalyNotificationDTO = new AnomalySubscriptionGroupNotificationDTO();
+ new AnomalySubscriptionGroupNotificationDTO();
+ anomalyNotificationDTO.setAnomalyId(anomaly.getId());
+ anomalyNotificationDTO.setDetectionConfigId(anomaly.getDetectionConfigId());
+ } else {
+ // update the existing record if the anomaly needs to be re-notified
+ anomalyNotificationDTO = subscriptionGroupNotificationDTOs.get(0);
+ anomalyNotificationDTO.setNotifiedSubscriptionGroupIds(Collections.emptyList());
+ }
+ this.anomalySubscriptionGroupNotificationDAO.save(anomalyNotificationDTO);
+ }
+ }
+
ThirdeyeMetricsUtil.detectionTaskSuccessCounter.inc();
LOG.info("End detection for config {} between {} and {}. Detected {} anomalies.", config.getId(), info.start,
info.end, result.getAnomalies());
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
index c0fdb51..f94f700 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
@@ -123,6 +123,7 @@ public class DetectionResource {
private final DetectionAlertConfigFormatter subscriptionConfigFormatter;
private final AggregationLoader aggregationLoader;
private final DetectionConfigurationResource detectionConfigurationResource;
+ private final AnomalySubscriptionGroupNotificationManager anomalySubscriptionGroupNotificationManager;
@Inject
public DetectionResource(
@@ -137,6 +138,7 @@ public class DetectionResource {
this.detectionAlertConfigDAO = DAORegistry.getInstance().getDetectionAlertConfigManager();
this.evaluationDAO = DAORegistry.getInstance().getEvaluationManager();
this.taskDAO = DAORegistry.getInstance().getTaskDAO();
+ this.anomalySubscriptionGroupNotificationManager = DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
TimeSeriesLoader timeseriesLoader =
new DefaultTimeSeriesLoader(metricDAO, datasetDAO, ThirdEyeCacheRegistry.getInstance().getQueryCache(), ThirdEyeCacheRegistry.getInstance().getTimeSeriesCache());
@@ -573,14 +575,18 @@ public class DetectionResource {
return Response.ok(health).build();
}
- @GET
- @Path(value = "/alert")
- public Response alert() {
- AnomalySubscriptionGroupNotificationManager anomalySubscriptionGroupNotificationManager = DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
- AnomalySubscriptionGroupNotificationDTO anomalySubscriptionGroupNotificationDTO = new AnomalySubscriptionGroupNotificationDTO();
- anomalySubscriptionGroupNotificationDTO.setAnomalyId(1L);
- anomalySubscriptionGroupNotificationDTO.setDetectionConfigId(2L);
- anomalySubscriptionGroupNotificationManager.save(anomalySubscriptionGroupNotificationDTO);
+ @POST
+ @Path(value = "/re-notify")
+ @ApiOperation("Resend the notification for the anomalies to the subscribed notification groups, if the subscription group supports re-notify")
+ public Response alert(@QueryParam("id") List<Long> anomalyIds) {
+ List<MergedAnomalyResultDTO> anomalies = this.anomalyDAO.findByIds(anomalyIds);
+ for (MergedAnomalyResultDTO anomaly : anomalies) {
+ AnomalySubscriptionGroupNotificationDTO anomalySubscriptionGroupNotificationDTO =
+ new AnomalySubscriptionGroupNotificationDTO();
+ anomalySubscriptionGroupNotificationDTO.setAnomalyId(anomaly.getId());
+ anomalySubscriptionGroupNotificationDTO.setDetectionConfigId(anomaly.getDetectionConfigId());
+ anomalySubscriptionGroupNotificationManager.save(anomalySubscriptionGroupNotificationDTO);
+ }
return Response.ok().build();
}
}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/DetectionAlertJob.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/DetectionAlertJob.java
index 70b318c..4026869 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/DetectionAlertJob.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/DetectionAlertJob.java
@@ -24,9 +24,11 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;
import java.util.Map;
import org.apache.pinot.thirdeye.anomaly.task.TaskConstants;
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
import org.apache.pinot.thirdeye.datalayer.bao.DetectionAlertConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.MergedAnomalyResultManager;
import org.apache.pinot.thirdeye.datalayer.bao.TaskManager;
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionAlertConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.TaskDTO;
import org.apache.pinot.thirdeye.datalayer.util.Predicate;
@@ -48,12 +50,14 @@ public class DetectionAlertJob implements Job {
private DetectionAlertConfigManager alertConfigDAO;
private TaskManager taskDAO;
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
- private MergedAnomalyResultManager anomalyDAO;
+ private final MergedAnomalyResultManager anomalyDAO;
+ private final AnomalySubscriptionGroupNotificationManager anomalySubscriptionGroupNotificationDAO;
public DetectionAlertJob() {
this.alertConfigDAO = DAORegistry.getInstance().getDetectionAlertConfigManager();
this.taskDAO = DAORegistry.getInstance().getTaskDAO();
this.anomalyDAO = DAORegistry.getInstance().getMergedAnomalyResultDAO();
+ this.anomalySubscriptionGroupNotificationDAO = DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
}
@Override
@@ -110,6 +114,8 @@ public class DetectionAlertJob implements Job {
* For example, if previous anomaly is from t1 to t2 generated at t3, then the timestamp in vectorLock is t3.
* If there is a new anomaly from t2 to t4 generated at t5, then we can still get this anomaly as t5 > t3.
*
+ * Also, check if there is any anomaly that needs re-notifying
+ *
* @param configDTO The Subscription Configuration.
* @return true if it needs notification task. false otherwise.
*/
@@ -123,6 +129,10 @@ public class DetectionAlertJob implements Job {
return true;
}
}
- return false;
+ // in addition to checking the watermarks, check if any anomalies need to be re-notified by querying the anomaly subscription group notification table
+ List<AnomalySubscriptionGroupNotificationDTO> anomalySubscriptionGroupNotifications =
+ this.anomalySubscriptionGroupNotificationDAO.findByPredicate(
+ Predicate.IN("detectionConfigId", vectorLocks.keySet().toArray()));
+ return !anomalySubscriptionGroupNotifications.isEmpty();
}
}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/filter/AnomalySeverityAlertFilter.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/filter/AnomalySeverityAlertFilter.java
new file mode 100644
index 0000000..1042ac7
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/alert/filter/AnomalySeverityAlertFilter.java
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.thirdeye.detection.alert.filter;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.pinot.thirdeye.constant.AnomalySeverity;
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.DetectionAlertConfigDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
+import org.apache.pinot.thirdeye.datalayer.util.Predicate;
+import org.apache.pinot.thirdeye.datasource.DAORegistry;
+import org.apache.pinot.thirdeye.detection.ConfigUtils;
+import org.apache.pinot.thirdeye.detection.DataProvider;
+import org.apache.pinot.thirdeye.detection.alert.DetectionAlertFilterNotification;
+import org.apache.pinot.thirdeye.detection.alert.DetectionAlertFilterResult;
+import org.apache.pinot.thirdeye.detection.alert.StatefulDetectionAlertFilter;
+import org.apache.pinot.thirdeye.detection.annotation.AlertFilter;
+
+
+/**
+ * The detection alert filter that can send notifications through multiple channels
+ * to a set of unconditional and another set of conditional recipients, based on the
+ * value of a specified anomaly severities.
+ *
+ * You can configure anomaly severities along with a variety of alerting
+ * channels and reference links.
+ *
+ * This alert pipeline have the capability of re-notify anomalies if the anomaly's severity is
+ * changed after it's created.
+ *
+ * <pre>
+ * severityRecipients:
+ * - severity:
+ * - LOW
+ * notify:
+ * jiraScheme:
+ * project: PROJECT
+ * assignee: oncall
+ * emailScheme:
+ * recipients:
+ * to:
+ * - "oncall@comany.com"
+ * - severity:
+ * - HIGH
+ * - CRITICAL
+ * notify:
+ * jiraScheme:
+ * project: PROJECT
+ * assignee: manager
+ * </pre>
+ */
+@AlertFilter(type = "SEVERITY_ALERTER_PIPELINE")
+public class AnomalySeverityAlertFilter extends StatefulDetectionAlertFilter {
+ public static final String PROP_DETECTION_CONFIG_IDS = "detectionConfigIds";
+ public static final String PROP_SEVERITY = "severity";
+ public static final String PROP_NOTIFY = "notify";
+ public static final String PROP_REF_LINKS = "referenceLinks";
+ public static final String PROP_SEVERITY_RECIPIENTS = "severityRecipients";
+
+ private final List<Map<String, Object>> severityRecipients;
+ private final List<Long> detectionConfigIds;
+
+ private final AnomalySubscriptionGroupNotificationManager anomalySubscriptionGroupNotificationDAO;
+
+
+ public AnomalySeverityAlertFilter(DataProvider provider, DetectionAlertConfigDTO config, long endTime) {
+ super(provider, config, endTime);
+ this.severityRecipients = ConfigUtils.getList(this.config.getProperties().get(PROP_SEVERITY_RECIPIENTS));
+ this.detectionConfigIds = ConfigUtils.getLongs(this.config.getProperties().get(PROP_DETECTION_CONFIG_IDS));
+ this.anomalySubscriptionGroupNotificationDAO =
+ DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
+ }
+
+ @Override
+ public DetectionAlertFilterResult run() {
+ DetectionAlertFilterResult result = new DetectionAlertFilterResult();
+
+ // retrieve the anomalies based on vector clocks
+ Set<MergedAnomalyResultDTO> anomalies = this.filter(this.makeVectorClocks(this.detectionConfigIds));
+ // find the anomalies that needs re-notifying.
+ anomalies.addAll(this.retrieveRenotifyAnomalies(this.detectionConfigIds));
+ // Prepare mapping from severity-recipients to anomalies
+ for (Map<String, Object> severityRecipient : this.severityRecipients) {
+ List<AnomalySeverity> severities = ConfigUtils.getList(severityRecipient.get(PROP_SEVERITY))
+ .stream()
+ .map(s -> AnomalySeverity.valueOf((String) s))
+ .collect(Collectors.toList());
+ Set<MergedAnomalyResultDTO> notifyAnomalies = new HashSet<>();
+ for (MergedAnomalyResultDTO anomaly : anomalies) {
+ if (severities.contains(anomaly.getSeverity())) {
+ notifyAnomalies.add(anomaly);
+ }
+ }
+
+ if (!notifyAnomalies.isEmpty()) {
+ DetectionAlertConfigDTO subsConfig = SubscriptionUtils.makeChildSubscriptionConfig(config,
+ ConfigUtils.getMap(severityRecipient.get(PROP_NOTIFY)),
+ ConfigUtils.getMap(severityRecipient.get(PROP_REF_LINKS)));
+ result.addMapping(new DetectionAlertFilterNotification(subsConfig), notifyAnomalies);
+ }
+ }
+
+ // Notify the remaining anomalies to default recipients
+ Set<MergedAnomalyResultDTO> allNotifiedAnomalies = new HashSet<>(result.getAllAnomalies());
+ Set<MergedAnomalyResultDTO> defaultAnomalies = new HashSet<>();
+ for (MergedAnomalyResultDTO anomaly : anomalies) {
+ if (!allNotifiedAnomalies.contains(anomaly)) {
+ defaultAnomalies.add(anomaly);
+ }
+ }
+ if (!defaultAnomalies.isEmpty()) {
+ result.addMapping(new DetectionAlertFilterNotification(config), defaultAnomalies);
+ }
+
+ return result;
+ }
+
+
+ protected Collection<MergedAnomalyResultDTO> retrieveRenotifyAnomalies(Collection<Long> detectionConfigIds) {
+ // find if any notification is needed
+ List<AnomalySubscriptionGroupNotificationDTO> anomalySubscriptionGroupNotificationDTOs =
+ this.anomalySubscriptionGroupNotificationDAO.findByPredicate(
+ Predicate.IN("detectionConfigId", detectionConfigIds.toArray()));
+
+ List<Long> anomalyIds = new ArrayList<>();
+ for (AnomalySubscriptionGroupNotificationDTO anomalySubscriptionGroupNotification : anomalySubscriptionGroupNotificationDTOs) {
+ // notify the anomalies if this subscription group have not sent out this anomaly yet
+ if (!anomalySubscriptionGroupNotification.getNotifiedSubscriptionGroupIds().contains(this.config.getId())) {
+ anomalyIds.add(anomalySubscriptionGroupNotification.getAnomalyId());
+ // add this subscription group to the notification record and update
+ anomalySubscriptionGroupNotification.getNotifiedSubscriptionGroupIds().add(this.config.getId());
+ this.anomalySubscriptionGroupNotificationDAO.save(anomalySubscriptionGroupNotification);
+ }
+ }
+ return anomalyIds.isEmpty() ? Collections.emptyList()
+ : DAORegistry.getInstance().getMergedAnomalyResultDAO().findByIds(anomalyIds);
+ }
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/SubscriptionConfigTranslator.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/SubscriptionConfigTranslator.java
index 17cad01..3f0e2aa 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/SubscriptionConfigTranslator.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/yaml/translator/SubscriptionConfigTranslator.java
@@ -69,10 +69,11 @@ public class SubscriptionConfigTranslator extends ConfigTranslator<DetectionAler
private static final String PROP_DIMENSION = "dimension";
private static final String PROP_DIMENSION_RECIPIENTS = "dimensionRecipients";
+ private static final String PROP_SEVERITY_RECIPIENTS = "severityRecipients";
private static final DetectionAlertRegistry DETECTION_ALERT_REGISTRY = DetectionAlertRegistry.getInstance();
private static final Set<String> PROPERTY_KEYS = new HashSet<>(
- Arrays.asList(PROP_RECIPIENTS, PROP_DIMENSION, PROP_DIMENSION_RECIPIENTS));
+ Arrays.asList(PROP_RECIPIENTS, PROP_DIMENSION, PROP_DIMENSION_RECIPIENTS, PROP_SEVERITY_RECIPIENTS));
private final DetectionConfigManager detectionConfigDAO = DAORegistry.getInstance().getDetectionConfigManager();
diff --git a/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AlertFilterUtils.java b/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AlertFilterUtils.java
index a1091cf..72f64db 100644
--- a/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AlertFilterUtils.java
+++ b/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AlertFilterUtils.java
@@ -26,6 +26,7 @@ import java.util.Map;
import java.util.Set;
import org.apache.pinot.thirdeye.anomaly.AnomalyType;
import org.apache.pinot.thirdeye.common.dimension.DimensionMap;
+import org.apache.pinot.thirdeye.constant.AnomalySeverity;
import org.apache.pinot.thirdeye.datalayer.dto.AnomalyFeedbackDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionAlertConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
@@ -89,9 +90,14 @@ public class AlertFilterUtils {
return new DetectionAlertFilterNotification(subsConfig);
}
-
static MergedAnomalyResultDTO makeAnomaly(Long configId, long baseTime, long start, long end,
Map<String, String> dimensions, AnomalyFeedbackDTO feedback) {
+ return makeAnomaly(configId, baseTime, start, end, dimensions, feedback, AnomalySeverity.DEFAULT);
+ }
+
+
+ static MergedAnomalyResultDTO makeAnomaly(Long configId, long baseTime, long start, long end,
+ Map<String, String> dimensions, AnomalyFeedbackDTO feedback, AnomalySeverity severity) {
MergedAnomalyResultDTO anomaly = DetectionTestUtils.makeAnomaly(configId, baseTime + start, baseTime + end);
anomaly.setType(AnomalyType.DEVIATION);
anomaly.setChildIds(Collections.emptySet());
@@ -108,6 +114,7 @@ public class AlertFilterUtils {
anomaly.setCreatedBy("no-auth-user");
anomaly.setUpdatedBy("no-auth-user");
+ anomaly.setSeverity(severity);
anomaly.setId(DAORegistry.getInstance().getMergedAnomalyResultDAO().save(anomaly));
if (feedback != null) {
diff --git a/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AnomalySeverityAlertFilterTest.java b/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AnomalySeverityAlertFilterTest.java
new file mode 100644
index 0000000..38f9189
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/detection/alert/filter/AnomalySeverityAlertFilterTest.java
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.detection.alert.filter;
+
+import com.google.common.collect.ImmutableMap;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.thirdeye.constant.AnomalySeverity;
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
+import org.apache.pinot.thirdeye.datalayer.bao.DAOTestBase;
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.DetectionAlertConfigDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.DetectionConfigDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
+import org.apache.pinot.thirdeye.datasource.DAORegistry;
+import org.apache.pinot.thirdeye.detection.MockDataProvider;
+import org.apache.pinot.thirdeye.detection.alert.DetectionAlertFilter;
+import org.apache.pinot.thirdeye.detection.alert.DetectionAlertFilterNotification;
+import org.apache.pinot.thirdeye.detection.alert.DetectionAlertFilterResult;
+import org.testng.Assert;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.apache.pinot.thirdeye.detection.alert.filter.AlertFilterUtils.*;
+
+
+public class AnomalySeverityAlertFilterTest {
+ private static final String PROP_RECIPIENTS = "recipients";
+ private static final String PROP_EMAIL_SCHEME = "emailScheme";
+ private static final String PROP_TO = "to";
+ private static final String PROP_CC = "cc";
+ private static final String PROP_BCC = "bcc";
+ private static final Set<String> PROP_TO_FOR_VALUE =
+ new HashSet<>(Arrays.asList("myTest@example.com", "myTest@example.org"));
+ private static final Set<String> PROP_TO_FOR_ANOTHER_VALUE =
+ new HashSet<>(Arrays.asList("myTest@example.net", "myTest@example.com"));
+ private static final String PROP_DETECTION_CONFIG_IDS = "detectionConfigIds";
+ private static final String PROP_SEVERITY_TO = "severityRecipients";
+ private static final List<Object> severityProperty = new ArrayList<>();
+
+ private DetectionAlertFilter alertFilter;
+
+ private DAOTestBase testDAOProvider;
+ private DetectionAlertConfigDTO alertConfig;
+ private List<MergedAnomalyResultDTO> detectionAnomalies;
+ private long baseTime;
+ private List<Long> detectionConfigIds;
+ private MergedAnomalyResultDTO renotifyAnomaly;
+ private final MockDataProvider provider = new MockDataProvider();
+ private final Map<String, Object> notify1 = new HashMap<>();
+ private final Map<String, Object> notify2 = new HashMap<>();
+ private final Map<String, Object> defaultScheme = new HashMap<>();
+
+ @BeforeMethod
+ public void beforeMethod() throws InterruptedException {
+ testDAOProvider = DAOTestBase.getInstance();
+
+ DetectionConfigDTO detectionConfig1 = new DetectionConfigDTO();
+ detectionConfig1.setName("test detection 1");
+ detectionConfig1.setActive(true);
+ long detectionConfigId1 = DAORegistry.getInstance().getDetectionConfigManager().save(detectionConfig1);
+
+ DetectionConfigDTO detectionConfig2 = new DetectionConfigDTO();
+ detectionConfig2.setName("test detection 2");
+ detectionConfig2.setActive(true);
+ long detectionConfigId2 = DAORegistry.getInstance().getDetectionConfigManager().save(detectionConfig2);
+
+ detectionConfigIds = Arrays.asList(detectionConfigId1, detectionConfigId2);
+
+ // Anomaly notification is tracked through create time. Start and end time doesn't matter here.
+ this.detectionAnomalies = new ArrayList<>();
+ renotifyAnomaly =
+ makeAnomaly(detectionConfigId1, System.currentTimeMillis(), 0, 50, Collections.singletonMap("key", "value"),
+ null, AnomalySeverity.LOW);
+ Thread.sleep(100);
+ this.baseTime = System.currentTimeMillis();
+ Thread.sleep(100);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId1, this.baseTime, 0, 100, Collections.singletonMap("key", "value"), null,
+ AnomalySeverity.LOW));
+ Thread.sleep(10);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId1, this.baseTime, 0, 110, Collections.singletonMap("key", "anotherValue"), null,
+ AnomalySeverity.MODERATE));
+ Thread.sleep(20);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId1, this.baseTime, 0, 120, Collections.singletonMap("key", "unknownValue"), null,
+ AnomalySeverity.HIGH));
+ Thread.sleep(30);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId2, this.baseTime, 110, 150, Collections.singletonMap("unknownKey", "value"),
+ null));
+ Thread.sleep(10);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId2, this.baseTime, 120, 160, Collections.singletonMap("key", "value"), null));
+ Thread.sleep(40);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId1, this.baseTime, 150, 200, Collections.<String, String>emptyMap(), null));
+ Thread.sleep(200);
+ this.detectionAnomalies.add(
+ makeAnomaly(detectionConfigId2, this.baseTime, 300, 400, Collections.singletonMap("key", "value"), null));
+ Thread.sleep(100);
+
+ this.alertConfig = createDetectionAlertConfig();
+ }
+
+ private DetectionAlertConfigDTO createDetectionAlertConfig() {
+ DetectionAlertConfigDTO alertConfig = new DetectionAlertConfigDTO();
+
+ notify1.put("severity", Arrays.asList("LOW", "MODERATE"));
+ notify1.put("notify", ImmutableMap.of("emailScheme", ImmutableMap.of("recipients", PROP_TO_FOR_VALUE)));
+ notify2.put("severity", Collections.singleton("HIGH"));
+ notify2.put("notify", ImmutableMap.of("emailScheme", ImmutableMap.of("recipients", PROP_TO_FOR_ANOTHER_VALUE)));
+ severityProperty.add(notify1);
+ severityProperty.add(notify2);
+
+ Map<String, Object> properties = new HashMap<>();
+ properties.put(PROP_DETECTION_CONFIG_IDS, detectionConfigIds);
+ properties.put(PROP_SEVERITY_TO, severityProperty);
+ alertConfig.setProperties(properties);
+
+ Map<String, Object> emailScheme = new HashMap<>();
+ Map<String, Set<String>> recipients = new HashMap<>();
+ recipients.put(PROP_TO, AlertFilterUtils.PROP_TO_VALUE);
+ recipients.put(PROP_CC, AlertFilterUtils.PROP_CC_VALUE);
+ recipients.put(PROP_BCC, AlertFilterUtils.PROP_BCC_VALUE);
+ emailScheme.put(PROP_RECIPIENTS, recipients);
+ defaultScheme.put(PROP_EMAIL_SCHEME, emailScheme);
+ alertConfig.setAlertSchemes(defaultScheme);
+
+ Map<Long, Long> vectorClocks = new HashMap<>();
+ vectorClocks.put(detectionConfigIds.get(0), this.baseTime);
+ vectorClocks.put(detectionConfigIds.get(1), this.baseTime);
+ alertConfig.setVectorClocks(vectorClocks);
+
+ return alertConfig;
+ }
+
+ @Test
+ public void testAlertFilterRecipients() throws Exception {
+ this.alertFilter = new AnomalySeverityAlertFilter(provider, alertConfig, this.baseTime + 350L);
+
+ DetectionAlertFilterResult result = this.alertFilter.run();
+ Assert.assertEquals(result.getResult().size(), 3);
+
+ int verifiedResult = 0;
+ for (Map.Entry<DetectionAlertFilterNotification, Set<MergedAnomalyResultDTO>> entry : result.getResult()
+ .entrySet()) {
+ if (entry.getValue().equals(makeSet(0, 1))) {
+ Assert.assertEquals(entry.getKey().getSubscriptionConfig().getAlertSchemes(), notify1.get("notify"));
+ verifiedResult++;
+ } else if (entry.getValue().equals(makeSet(2))) {
+ Assert.assertEquals(entry.getKey().getSubscriptionConfig().getAlertSchemes(), notify2.get("notify"));
+ verifiedResult++;
+ } else if (entry.getValue().equals(makeSet(3, 4, 5))) {
+ Assert.assertEquals(entry.getKey().getSubscriptionConfig().getAlertSchemes(), defaultScheme);
+ verifiedResult++;
+ }
+ }
+ Assert.assertEquals(verifiedResult, 3);
+ }
+
+ @Test
+ public void testRenotifyAnomaly() throws Exception {
+ AnomalySubscriptionGroupNotificationManager renotificationManager =
+ DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
+ AnomalySubscriptionGroupNotificationDTO anomalySubscriptionGroupNotification =
+ new AnomalySubscriptionGroupNotificationDTO();
+ anomalySubscriptionGroupNotification.setAnomalyId(renotifyAnomaly.getId());
+ anomalySubscriptionGroupNotification.setDetectionConfigId(renotifyAnomaly.getDetectionConfigId());
+ renotificationManager.save(anomalySubscriptionGroupNotification);
+
+ this.alertFilter = new AnomalySeverityAlertFilter(provider, alertConfig, this.baseTime + 350L);
+
+ DetectionAlertFilterResult result = this.alertFilter.run();
+ Assert.assertEquals(result.getResult().size(), 3);
+
+ int verifiedResult = 0;
+ for (Map.Entry<DetectionAlertFilterNotification, Set<MergedAnomalyResultDTO>> entry : result.getResult()
+ .entrySet()) {
+ if (entry.getValue().equals(makeSet(renotifyAnomaly, 0, 1))) {
+ Assert.assertEquals(entry.getKey().getSubscriptionConfig().getAlertSchemes(), notify1.get("notify"));
+ verifiedResult++;
+ } else if (entry.getValue().equals(makeSet(2))) {
+ Assert.assertEquals(entry.getKey().getSubscriptionConfig().getAlertSchemes(), notify2.get("notify"));
+ verifiedResult++;
+ } else if (entry.getValue().equals(makeSet(3, 4, 5))) {
+ Assert.assertEquals(entry.getKey().getSubscriptionConfig().getAlertSchemes(), defaultScheme);
+ verifiedResult++;
+ }
+ }
+ Assert.assertEquals(verifiedResult, 3);
+ }
+
+ private Set<MergedAnomalyResultDTO> makeSet(MergedAnomalyResultDTO anomaly, int... anomalyIndices) {
+ Set<MergedAnomalyResultDTO> set = makeSet(anomalyIndices);
+ set.add(anomaly);
+ return set;
+ }
+
+ private Set<MergedAnomalyResultDTO> makeSet(int... anomalyIndices) {
+ Set<MergedAnomalyResultDTO> output = new HashSet<>();
+ for (int anomalyIndex : anomalyIndices) {
+ output.add(this.detectionAnomalies.get(anomalyIndex));
+ }
+ return output;
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/03: add anomaly notification table
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch severity-alerter
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 8bc3d4854c806289467cdfb65ab8b17ad8a65cb2
Author: Jihao Zhang <ji...@linkedin.com>
AuthorDate: Fri Aug 21 15:23:17 2020 -0700
add anomaly notification table
---
...nomalySubscriptionGroupNotificationManager.java | 28 ++++++++
...lySubscriptionGroupNotificationManagerImpl.java | 34 +++++++++
.../thirdeye/datalayer/dao/GenericPojoDao.java | 4 ++
.../AnomalySubscriptionGroupNotificationDTO.java | 27 +++++++
.../AnomalySubscriptionGroupNotificationIndex.java | 42 +++++++++++
.../AnomalySubscriptionGroupNotificationBean.java | 82 ++++++++++++++++++++++
.../thirdeye/datalayer/util/DaoProviderUtil.java | 3 +
.../pinot/thirdeye/datasource/DAORegistry.java | 6 ++
.../thirdeye/detection/DetectionResource.java | 13 ++++
.../src/main/resources/schema/create-schema.sql | 12 ++++
10 files changed, 251 insertions(+)
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/AnomalySubscriptionGroupNotificationManager.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/AnomalySubscriptionGroupNotificationManager.java
new file mode 100644
index 0000000..28afb75
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/AnomalySubscriptionGroupNotificationManager.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.datalayer.bao;
+
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
+
+
+public interface AnomalySubscriptionGroupNotificationManager
+ extends AbstractManager<AnomalySubscriptionGroupNotificationDTO> {
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/jdbc/AnomalySubscriptionGroupNotificationManagerImpl.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/jdbc/AnomalySubscriptionGroupNotificationManagerImpl.java
new file mode 100644
index 0000000..2aa434c
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/bao/jdbc/AnomalySubscriptionGroupNotificationManagerImpl.java
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.datalayer.bao.jdbc;
+
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
+import org.apache.pinot.thirdeye.datalayer.pojo.AnomalySubscriptionGroupNotificationBean;
+
+
+public class AnomalySubscriptionGroupNotificationManagerImpl
+ extends AbstractManagerImpl<AnomalySubscriptionGroupNotificationDTO>
+ implements AnomalySubscriptionGroupNotificationManager {
+ public AnomalySubscriptionGroupNotificationManagerImpl() {
+ super(AnomalySubscriptionGroupNotificationDTO.class, AnomalySubscriptionGroupNotificationBean.class);
+ }
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dao/GenericPojoDao.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dao/GenericPojoDao.java
index 4edc4a5..1ad80d0 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dao/GenericPojoDao.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dao/GenericPojoDao.java
@@ -33,6 +33,7 @@ import org.apache.pinot.thirdeye.datalayer.entity.AlertConfigIndex;
import org.apache.pinot.thirdeye.datalayer.entity.AlertSnapshotIndex;
import org.apache.pinot.thirdeye.datalayer.entity.AnomalyFeedbackIndex;
import org.apache.pinot.thirdeye.datalayer.entity.AnomalyFunctionIndex;
+import org.apache.pinot.thirdeye.datalayer.entity.AnomalySubscriptionGroupNotificationIndex;
import org.apache.pinot.thirdeye.datalayer.entity.ApplicationIndex;
import org.apache.pinot.thirdeye.datalayer.entity.ClassificationConfigIndex;
import org.apache.pinot.thirdeye.datalayer.entity.ConfigIndex;
@@ -62,6 +63,7 @@ import org.apache.pinot.thirdeye.datalayer.pojo.AlertConfigBean;
import org.apache.pinot.thirdeye.datalayer.pojo.AlertSnapshotBean;
import org.apache.pinot.thirdeye.datalayer.pojo.AnomalyFeedbackBean;
import org.apache.pinot.thirdeye.datalayer.pojo.AnomalyFunctionBean;
+import org.apache.pinot.thirdeye.datalayer.pojo.AnomalySubscriptionGroupNotificationBean;
import org.apache.pinot.thirdeye.datalayer.pojo.ApplicationBean;
import org.apache.pinot.thirdeye.datalayer.pojo.ClassificationConfigBean;
import org.apache.pinot.thirdeye.datalayer.pojo.ConfigBean;
@@ -171,6 +173,8 @@ public class GenericPojoDao {
newPojoInfo(DEFAULT_BASE_TABLE_NAME, RootcauseTemplateIndex.class));
pojoInfoMap.put(OnlineDetectionDataBean.class,
newPojoInfo(DEFAULT_BASE_TABLE_NAME, OnlineDetectionDataIndex.class));
+ pojoInfoMap.put(AnomalySubscriptionGroupNotificationBean.class,
+ newPojoInfo(DEFAULT_BASE_TABLE_NAME, AnomalySubscriptionGroupNotificationIndex.class));
}
private static PojoInfo newPojoInfo(String baseTableName,
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/AnomalySubscriptionGroupNotificationDTO.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/AnomalySubscriptionGroupNotificationDTO.java
new file mode 100644
index 0000000..f1834a3
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/dto/AnomalySubscriptionGroupNotificationDTO.java
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.datalayer.dto;
+
+import org.apache.pinot.thirdeye.datalayer.pojo.AnomalySubscriptionGroupNotificationBean;
+
+
+public class AnomalySubscriptionGroupNotificationDTO extends AnomalySubscriptionGroupNotificationBean {
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/entity/AnomalySubscriptionGroupNotificationIndex.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/entity/AnomalySubscriptionGroupNotificationIndex.java
new file mode 100644
index 0000000..8bd68b1
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/entity/AnomalySubscriptionGroupNotificationIndex.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.datalayer.entity;
+
+public class AnomalySubscriptionGroupNotificationIndex extends AbstractIndexEntity {
+ long anomalyId;
+ long detectionConfigId;
+
+ public long getAnomalyId() {
+ return anomalyId;
+ }
+
+ public void setAnomalyId(long anomalyId) {
+ this.anomalyId = anomalyId;
+ }
+
+ public long getDetectionConfigId() {
+ return detectionConfigId;
+ }
+
+ public void setDetectionConfigId(long detectionConfigId) {
+ this.detectionConfigId = detectionConfigId;
+ }
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/AnomalySubscriptionGroupNotificationBean.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/AnomalySubscriptionGroupNotificationBean.java
new file mode 100644
index 0000000..4d7413e
--- /dev/null
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/pojo/AnomalySubscriptionGroupNotificationBean.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ */
+
+package org.apache.pinot.thirdeye.datalayer.pojo;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+
+@JsonIgnoreProperties(ignoreUnknown=true)
+public class AnomalySubscriptionGroupNotificationBean extends AbstractBean {
+ private Long anomalyId;
+ private Long detectionConfigId;
+ private List<Long> notifiedSubscriptionGroupIds = new ArrayList<>();
+
+ public Long getAnomalyId() {
+ return anomalyId;
+ }
+
+ public void setAnomalyId(Long anomalyId) {
+ this.anomalyId = anomalyId;
+ }
+
+ public Long getDetectionConfigId() {
+ return detectionConfigId;
+ }
+
+ public void setDetectionConfigId(Long detectionConfigId) {
+ this.detectionConfigId = detectionConfigId;
+ }
+
+ public List<Long> getNotifiedSubscriptionGroupIds() {
+ return notifiedSubscriptionGroupIds;
+ }
+
+ public void setNotifiedSubscriptionGroupIds(List<Long> notifiedSubscriptionGroupIds) {
+ this.notifiedSubscriptionGroupIds = notifiedSubscriptionGroupIds;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ AnomalySubscriptionGroupNotificationBean that = (AnomalySubscriptionGroupNotificationBean) o;
+ return Objects.equals(anomalyId, that.anomalyId) && Objects.equals(detectionConfigId, that.detectionConfigId)
+ && Objects.equals(notifiedSubscriptionGroupIds, that.notifiedSubscriptionGroupIds);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(anomalyId, detectionConfigId, notifiedSubscriptionGroupIds);
+ }
+
+ @Override
+ public String toString() {
+ return "AnomalySubscriptionGroupNotificationBean{" + "anomalyId=" + anomalyId + ", detectionConfigId="
+ + detectionConfigId + ", notifiedSubscriptionGroupIds=" + notifiedSubscriptionGroupIds + '}';
+ }
+}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/util/DaoProviderUtil.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/util/DaoProviderUtil.java
index b244646..2c1cba6 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/util/DaoProviderUtil.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datalayer/util/DaoProviderUtil.java
@@ -30,6 +30,7 @@ import org.apache.pinot.thirdeye.datalayer.entity.AlertConfigIndex;
import org.apache.pinot.thirdeye.datalayer.entity.AlertSnapshotIndex;
import org.apache.pinot.thirdeye.datalayer.entity.AnomalyFeedbackIndex;
import org.apache.pinot.thirdeye.datalayer.entity.AnomalyFunctionIndex;
+import org.apache.pinot.thirdeye.datalayer.entity.AnomalySubscriptionGroupNotificationIndex;
import org.apache.pinot.thirdeye.datalayer.entity.ApplicationIndex;
import org.apache.pinot.thirdeye.datalayer.entity.ClassificationConfigIndex;
import org.apache.pinot.thirdeye.datalayer.entity.ConfigIndex;
@@ -209,6 +210,8 @@ public abstract class DaoProviderUtil {
convertCamelCaseToUnderscore(RootcauseTemplateIndex.class.getSimpleName()));
entityMappingHolder.register(conn, OnlineDetectionDataIndex.class,
convertCamelCaseToUnderscore(OnlineDetectionDataIndex.class.getSimpleName()));
+ entityMappingHolder.register(conn, AnomalySubscriptionGroupNotificationIndex.class,
+ convertCamelCaseToUnderscore(AnomalySubscriptionGroupNotificationIndex.class.getSimpleName()));
} catch (Exception e) {
throw new RuntimeException(e);
}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datasource/DAORegistry.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datasource/DAORegistry.java
index 1e48779..b7c5b56 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datasource/DAORegistry.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datasource/DAORegistry.java
@@ -22,6 +22,7 @@ package org.apache.pinot.thirdeye.datasource;
import org.apache.pinot.thirdeye.datalayer.bao.AlertConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.AlertSnapshotManager;
import org.apache.pinot.thirdeye.datalayer.bao.AnomalyFunctionManager;
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
import org.apache.pinot.thirdeye.datalayer.bao.ApplicationManager;
import org.apache.pinot.thirdeye.datalayer.bao.ClassificationConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.ConfigManager;
@@ -48,6 +49,7 @@ import org.apache.pinot.thirdeye.datalayer.bao.TaskManager;
import org.apache.pinot.thirdeye.datalayer.bao.jdbc.AlertConfigManagerImpl;
import org.apache.pinot.thirdeye.datalayer.bao.jdbc.AlertSnapshotManagerImpl;
import org.apache.pinot.thirdeye.datalayer.bao.jdbc.AnomalyFunctionManagerImpl;
+import org.apache.pinot.thirdeye.datalayer.bao.jdbc.AnomalySubscriptionGroupNotificationManagerImpl;
import org.apache.pinot.thirdeye.datalayer.bao.jdbc.ApplicationManagerImpl;
import org.apache.pinot.thirdeye.datalayer.bao.jdbc.ClassificationConfigManagerImpl;
import org.apache.pinot.thirdeye.datalayer.bao.jdbc.ConfigManagerImpl;
@@ -202,4 +204,8 @@ public class DAORegistry {
public OnlineDetectionDataManager getOnlineDetectionDataManager() {
return DaoProviderUtil.getInstance(OnlineDetectionDataManagerImpl.class);
}
+
+ public AnomalySubscriptionGroupNotificationManager getAnomalySubscriptionGroupNotificationManager() {
+ return DaoProviderUtil.getInstance(AnomalySubscriptionGroupNotificationManagerImpl.class);
+ }
}
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
index 6db047b..c0fdb51 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionResource.java
@@ -55,6 +55,7 @@ import org.apache.pinot.thirdeye.dashboard.resources.v2.ResourceUtils;
import org.apache.pinot.thirdeye.dashboard.resources.v2.rootcause.AnomalyEventFormatter;
import org.apache.pinot.thirdeye.dataframe.DataFrame;
import org.apache.pinot.thirdeye.dataframe.util.MetricSlice;
+import org.apache.pinot.thirdeye.datalayer.bao.AnomalySubscriptionGroupNotificationManager;
import org.apache.pinot.thirdeye.datalayer.bao.DatasetConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.DetectionAlertConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.DetectionConfigManager;
@@ -64,6 +65,7 @@ import org.apache.pinot.thirdeye.datalayer.bao.MergedAnomalyResultManager;
import org.apache.pinot.thirdeye.datalayer.bao.MetricConfigManager;
import org.apache.pinot.thirdeye.datalayer.bao.TaskManager;
import org.apache.pinot.thirdeye.datalayer.dto.AnomalyFeedbackDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.AnomalySubscriptionGroupNotificationDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DatasetConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionAlertConfigDTO;
import org.apache.pinot.thirdeye.datalayer.dto.DetectionConfigDTO;
@@ -570,4 +572,15 @@ public class DetectionResource {
}
return Response.ok(health).build();
}
+
+ @GET
+ @Path(value = "/alert")
+ public Response alert() {
+ AnomalySubscriptionGroupNotificationManager anomalySubscriptionGroupNotificationManager = DAORegistry.getInstance().getAnomalySubscriptionGroupNotificationManager();
+ AnomalySubscriptionGroupNotificationDTO anomalySubscriptionGroupNotificationDTO = new AnomalySubscriptionGroupNotificationDTO();
+ anomalySubscriptionGroupNotificationDTO.setAnomalyId(1L);
+ anomalySubscriptionGroupNotificationDTO.setDetectionConfigId(2L);
+ anomalySubscriptionGroupNotificationManager.save(anomalySubscriptionGroupNotificationDTO);
+ return Response.ok().build();
+ }
}
diff --git a/thirdeye/thirdeye-pinot/src/main/resources/schema/create-schema.sql b/thirdeye/thirdeye-pinot/src/main/resources/schema/create-schema.sql
index 8fe2c4f..ce0d7e8 100644
--- a/thirdeye/thirdeye-pinot/src/main/resources/schema/create-schema.sql
+++ b/thirdeye/thirdeye-pinot/src/main/resources/schema/create-schema.sql
@@ -449,3 +449,15 @@ create table if not exists online_detection_data_index (
create index online_detection_data_id_idx ON online_detection_data_index(base_id);
create index online_detection_data_dataset_idx ON online_detection_data_index(dataset);
create index online_detection_data_metric_idx ON online_detection_data_index(metric);
+
+create table if not exists anomaly_subscription_group_notification_index (
+ base_id bigint(20) not null,
+ anomaly_id bigint(20) not null,
+ detection_config_id bigint(20) not null,
+ create_time timestamp default 0,
+ update_time timestamp default current_timestamp,
+ version int(10)
+) ENGINE=InnoDB;
+ALTER TABLE `anomaly_subscription_group_notification_index` ADD UNIQUE `anomaly_subscription_group_notification_index`(anomaly_id);
+create index anomaly_subscription_group_anomaly_idx ON anomaly_subscription_group_notification_index(anomaly_id);
+create index anomaly_subscription_group_detection_config_idx ON anomaly_subscription_group_notification_index(anomaly_id)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 03/03: refactor
Posted by ji...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch severity-alerter
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 55117218c4008a8d5a3893f0b29892a52e5d9608
Author: Jihao Zhang <ji...@linkedin.com>
AuthorDate: Wed Sep 2 10:08:02 2020 -0700
refactor
---
.../detection/DetectionPipelineTaskRunner.java | 34 ++++++++++++----------
1 file changed, 19 insertions(+), 15 deletions(-)
diff --git a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java
index 183be6f..32bb024 100644
--- a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java
+++ b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/DetectionPipelineTaskRunner.java
@@ -179,21 +179,7 @@ public class DetectionPipelineTaskRunner implements TaskRunner {
for (MergedAnomalyResultDTO anomaly : result.getAnomalies()) {
// if an anomaly should be re-notified, update the notification lookup table in the database
if (anomaly.shouldRenotify()) {
- List<AnomalySubscriptionGroupNotificationDTO> subscriptionGroupNotificationDTOs =
- this.anomalySubscriptionGroupNotificationDAO.findByPredicate(Predicate.EQ("anomalyId", anomaly.getId()));
- AnomalySubscriptionGroupNotificationDTO anomalyNotificationDTO;
- if (subscriptionGroupNotificationDTOs.isEmpty()) {
- // create a new record if it is not existed yet.
- anomalyNotificationDTO = new AnomalySubscriptionGroupNotificationDTO();
- new AnomalySubscriptionGroupNotificationDTO();
- anomalyNotificationDTO.setAnomalyId(anomaly.getId());
- anomalyNotificationDTO.setDetectionConfigId(anomaly.getDetectionConfigId());
- } else {
- // update the existing record if the anomaly needs to be re-notified
- anomalyNotificationDTO = subscriptionGroupNotificationDTOs.get(0);
- anomalyNotificationDTO.setNotifiedSubscriptionGroupIds(Collections.emptyList());
- }
- this.anomalySubscriptionGroupNotificationDAO.save(anomalyNotificationDTO);
+ renotifyAnomaly(anomaly);
}
}
@@ -206,4 +192,22 @@ public class DetectionPipelineTaskRunner implements TaskRunner {
throw e;
}
}
+
+ private void renotifyAnomaly(MergedAnomalyResultDTO anomaly) {
+ List<AnomalySubscriptionGroupNotificationDTO> subscriptionGroupNotificationDTOs =
+ this.anomalySubscriptionGroupNotificationDAO.findByPredicate(Predicate.EQ("anomalyId", anomaly.getId()));
+ AnomalySubscriptionGroupNotificationDTO anomalyNotificationDTO;
+ if (subscriptionGroupNotificationDTOs.isEmpty()) {
+ // create a new record if it is not existed yet.
+ anomalyNotificationDTO = new AnomalySubscriptionGroupNotificationDTO();
+ new AnomalySubscriptionGroupNotificationDTO();
+ anomalyNotificationDTO.setAnomalyId(anomaly.getId());
+ anomalyNotificationDTO.setDetectionConfigId(anomaly.getDetectionConfigId());
+ } else {
+ // update the existing record if the anomaly needs to be re-notified
+ anomalyNotificationDTO = subscriptionGroupNotificationDTOs.get(0);
+ anomalyNotificationDTO.setNotifiedSubscriptionGroupIds(Collections.emptyList());
+ }
+ this.anomalySubscriptionGroupNotificationDAO.save(anomalyNotificationDTO);
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org