You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by ak...@apache.org on 2020/07/24 16:22:37 UTC
[incubator-pinot] branch master updated: [TE] Make few internal
documentationi and mocks publicwq (#5743)
This is an automated email from the ASF dual-hosted git repository.
akshayrai09 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new cb149be [TE] Make few internal documentationi and mocks publicwq (#5743)
cb149be is described below
commit cb149bed54b926b5841e7cc1919c5c3fc8dd8199
Author: Akshay Rai <ak...@linkedin.com>
AuthorDate: Fri Jul 24 09:22:17 2020 -0700
[TE] Make few internal documentationi and mocks publicwq (#5743)
---
thirdeye/docs/alert_setup.rst | 2 +-
thirdeye/docs/detection_pipeline_architecture.rst | 165 ++++++++++++++++++
.../docs/detection_pipeline_execution_flow.rst | 192 +++++++++++++++++++++
thirdeye/docs/index.rst | 2 +
.../{alert_setup.rst => thirdeye_architecture.rst} | 13 +-
.../{alert_setup.rst => thirdeye_ui_mocks.rst} | 15 +-
6 files changed, 372 insertions(+), 17 deletions(-)
diff --git a/thirdeye/docs/alert_setup.rst b/thirdeye/docs/alert_setup.rst
index bda73f0..be00ebe 100644
--- a/thirdeye/docs/alert_setup.rst
+++ b/thirdeye/docs/alert_setup.rst
@@ -29,4 +29,4 @@ Alert Setup
advanced_config
templates
appendix
- contribute_detection
\ No newline at end of file
+ contribute_detection
diff --git a/thirdeye/docs/detection_pipeline_architecture.rst b/thirdeye/docs/detection_pipeline_architecture.rst
new file mode 100644
index 0000000..0419b6c
--- /dev/null
+++ b/thirdeye/docs/detection_pipeline_architecture.rst
@@ -0,0 +1,165 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+.. _detection-pipeline-architecture:
+
+######################################
+Detection Pipeline Architecture
+######################################
+
+This document summarizes the motivation and rationale behind the 2018 refactoring of ThirdEye's anomaly framework library. We describe critical user, dev, and ops pain points, define a list of requirements and discuss our approach to building a scalable and robust detection framework for ThirdEye. We also discuss conscious design trade-offs made in order to facilitate future modification and maintenance of the system.
+
+Motivation
+########################
+ThirdEye has been adopted by numerous teams for production monitoring of business and system metrics, and ThirdEye's user base grows consistently. With this come challenges of scale and a demand for excellence in development and operation. ThirdEye has outgrown the assumptions and limits of its existing detection framework in multiple dimensions and we need to address technical debt in order to enable the on-boarding of new use-cases and continue to satisfy our existing customers. We exp [...]
+
+Lack of support for business rules and custom work flows
+***********************************************************
+Anomaly detection workflows have many common components such as the monitoring of different metrics and the drill-down into multiple dimensions of time series. However, each individual detection use-case usually comes with an additional set of business rules to integrate smoothly with established processes at LinkedIn. These typically include cut-off thresholds and fixed lists of sub-dimensions to monitor or ignore, but may extend to custom alerting workflows and the grouping of detected [...]
+
+Difficult debugging and modification
+**************************************
+The existing framework has grown over multiple generations of developers with different preferences, designs, and goals. This lead to an inconsistent and undocumented approach to architecture and code design. Worse, existing design documentation is outdated and misleading. This is exacerbated by a lack of testing infrastructure, integration tests, and unit tests. Many assumptions about the behavior of the framework are implicit and only exist in the heads of past maintainers of platform [...]
+
+Tight coupling prevents testing
+************************************
+Another property of the existing detection code base is tight coupling of individual components and the leaking of state in unexpected places, such as utility functions. It is not currently possible to unit test an individual piece, such as the implementation of an anomaly merger strategy, without setting up the entire system from caching and data layer, over scheduler and worker, to a populated database with pre-existing configuration and anomalies. The untangling of these dependencies [...]
+
+Low performance and limitations to scaling
+********************************************
+The current implementation of detection, on-boarding, and tuning executes slowly. This is surprising given the modest amount of time series ingested and processed by ThirdEye, especially when considering that Pinot and Autometrics do the heavy lifting of data aggregation and filtering. This can be attributed to the serial retrieval of data points and redundant data base access, part of which is a consequence of the many stacked generations of designs and code. The execution of on-boardin [...]
+
+Half-baked self-service and configuration capabilities
+********************************************************
+Configuration currently is specific to the chosen algorithm and no official front-end exists to manually modify detection settings. As a workaround users access and modify configuration directly via a database admin tool, which comes with its own set of dangers and pitfalls. While the database supports JSON serialization, the detection algorithms currently use their own custom string serialization format that is inaccessible even to advanced users. Additionally, the feedback cycle for us [...]
+
+No ensemble detection or multi-metric dependencies
+******************************************************
+Some users prefer execution of rules and algorithms in parallel, using algorithms for regular detection but relying on additional rules as fail-safe against false negatives. Also, detection cannot easily incorporate information from multiple metrics without encoding this functionality directly into the algorithm. For example, site-wide impact filters for business metrics are currently part of each separate algorithm implementation rather than modular, re-usable components.
+
+No sandbox support
+**********************
+On-boarding, performance tuning, and re-configuration of alerts are processes that involve iterative back-testing and to some degree rely on human judgement. In order for users to experiment with detection settings, the execution of algorithms and evaluation of rules must be sandboxed from affecting the state of the production system. Similarly, integration testing of new components may require parallel execution of production and staging environments to build trust after invasive change [...]
+
+Requirements
+################
+An architecture is only as concise as its requirements. After running ThirdEye for metric monitoring in production for over a year, many original assumptions changed and new requirements came in. In the following we summarize the requirements we deem critical for moving ThirdEye's detection capabilities onto a robust and scalable new foundation.
+
+De-coupling of components
+**************************************
+Components of the detection framework must be separated to a degree that allows testing of individual units and sandboxed execution of detection workflows. Furthermore, contracts (interfaces) between components should be minimal and should not pre-impose a structure that is modeled after specific use-cases.
+
+Full testability
+**************************************
+Every single part of the detection pipeline must be testable as a unit as well as in integration with others. This allows us to isolate problems a in individual components and avoid regressions via dedicated tests. We must alos provide test infrastructure to mock required components with simple implementations of existing interfaces. This testability requirement also serves as verification step of our efforts to decouple components.
+
+Gradual migration via emulation of existing anomaly interface
+****************************************************************************
+ThirdEye has an existing use-base that has built trust in existing detection methods and tweaked them to their needs, and hence, support for legacy algorithms via an emulation layer is a must-have. It is near impossible to ensure perfect consistency of legacy and emulated execution due to numerous undocumented behavioral quirks. Therefore, the emulation layer will be held to a minimum. Additionally, as we migrate users' workflows to newer implementations this support will be phased out.
+
+Simple algorithms should be simple to build, test, and configure
+*************************************************************************
+Simple algorithms and rules must be easy to implement, test, and configure. As a platform ThirdEye hosts different types of algorithms and continuously adds more. In order to scale development to both a larger team of developers and collaborators, development of custom workflows and algorithms must be kept as as friction-less as possible.
+
+Support multiple metrics and data sources in single algorithm
+*******************************************************************
+Several use-cases require information from several metrics and metric-dimensions at once to reliably detect and classify anomalies. Our framework needs native support for this integration of data from multiple sources. This includes multiple metrics, as well as other sources such as external events, past anomalies, etc.
+
+Use-case specific workflows and logic
+**************************************
+Most detection use-cases bring their own domain-specific business logic. These processes must be encoded into ThirdEye's detection workflows in order to integrate with existing processes at LinkedIn and enable the on-boarding of additional users and teams. This integration of business logic should be possible via configuration options in the majority of cases, but will eventually require additional plug-able code to execute during detection and alerting workflows.
+
+Don't repeat yourself (code and component re-use)
+****************************************************
+With often similar but not perfectly equal workflows there is a temptation to copy code sequences for existing algorithms and re-use them for new implementations. This redundancy leads to code bloat and the duplication of mistakes and should be avoided to the maximum degree possible. Code re-use via building blocks and utilities correctly designed to be stateless in nature must be a priority.
+
+Consistent configuration of algorithms (top-level and nested)
+******************************************************************
+The mechanism for algorithm configuration should be uniform across different implementations. This should be true also for nested algorithms. As ThirdEye already uses JSON as serialization format for data base storage, configuration should be stored in a compatible way. While we note JSON is not the best choice for human read-able configuration it is the straight-forward choice given the existing metadata infrastructure.
+
+Stateless, semi-stateful, stateful execution
+**********************************************
+Algorithms can exist in multiple environments. A stateless sandbox environment, a semi-stateful sandbox environment that has been prepared with data such as pre-existing anomalies, and the production environment in which the database keeps track of results of previous executions. The algorithm implementation should be oblivious to the executing harness to the maximum extent possible.
+
+Interactive detection preview and performance tuning for users
+*********************************************************************
+As part of the on-boarding workflow and tuning procedure, ThirdEye allows users to tweak settings - either manually or via automated parameter search. This functionality should support interactive replay and preview of detection results in order to help our users build trust in and improve on the detection algorithm or detection rules. This is primarily a performance requirement as is demands execution of newly-generated detection rules at user-interactive speeds. Furthermore, this inter [...]
+
+Flow parallelism
+***********************
+Multiple independent detection flows must execute in parallel and without affecting each other's results. Sub-task level parallelism is out of scope.
+
+Architecture
+#################
+We split the architecture of ThirdEye in three layers: the execution layer, the framework layer, and the logic layer. The execution layer is responsible for tying in the anomaly detection framework with ThirdEye's existing task execution framework and provide configuration facilities. The framework layer provides an abstraction for algorithm development by providing an unified interface for data and configuration retrieval as well as utilities for common aspects involved in algorithm dev [...]
+
+
+.. image:: https://user-images.githubusercontent.com/4448437/88264885-670d6500-cc81-11ea-92da-b69073a69e03.png
+ :width: 500
+
+Execution layer
+**********************
+The execution layer ties in the detection framework with ThirdEye's existing task execution framework and data layer. ThirdEye triggers the execution of detection algorithms either time-based (cron job) or on-demand for testing or per on-boarding request from a user. The scheduled execution executes per cron schedule in a stateful manner such that the result of previous executions is available to the algorithm on every run. This component is especially important as it serves most product [...]
+
+Framework layer
+*********************
+The framework provides an abstraction over various data sources and configuration facilities in ThirdEye and presents a uniform layer to pipeline and algorithm developers. A major aspect of this is the Data Provider, which encapsulates time-series, anomaly, and meta-data access. Furthermore, there are helpers for configuration injection and utilities for common aspects of algorithm development, such as time-series transformations and the data frame API. The framework layer also manages t [...]
+
+Logic layer
+*********************
+The business logic layer builds on the framework's pipeline contract to implement detection algorithms and specialized pipelines that share functionality across groups of similar algorithms. A special aspect of the business logic layer are wrapper pipelines which enable implementation and configuration of custom detection workflows, such as the exploration of individual dimensions or the domain-specific merging of anomalies with common dimensions. The framework pipelines supports this fu [...]
+
+Design decisions and trade-offs
+#####################################
+
+"Simple algorithms should be simple to build" vs "arbitrary workflows should be possible"
+***********************************************************************************************
+Our detection framework provides a layered pipeline API to balance simplicity and flexibility in algorithm and workflow development. We chose to provide two layers: the raw "DetectionPipeline" and the safer "StaticDetectionPipeline". The raw pipeline layer allows dynamic loading of data and iterative execution of nested code, which enables us to implement arbitrary workflows but comes at the cost higher complexity and placing the burden of performance optimization on the developer. The s [...]
+
+"De-coupling" vs "simple infrastructure"
+*******************************************
+Simplicity and testability stand at the core of the refactoring of the anomaly detection framework. De-coupling of components is strictly necessary to enable unit testing, however, a separation of the framework into dozens of individual components makes the writing of algorithms and test-cases confusing and difficult, especially as it introduces various partial-failure modes. The data provider shows this trade-off between loose coupling and one-stop simplicity: rather than registering in [...]
+
+"batch detection" vs "point-wise walk forward"
+*************************************************
+The detection pipeline contract was designed to handle time ranges rather than single timestamps. This enables batch operations and multi-period detection scenarios but offloads some complexity of implementing walk-forward analysis onto the maintainers of algorithms that perform point-wise anomaly detection. At the current state this is mainly an issue with legacy detection algorithms and we address it by providing a specialized wrapper pipeline that contains a generic implementation of [...]
+
+"complex algorithms" vs "performance and scalability"
+**********************************************************
+Our architecture currently does not enforce any structure on the algorithm implementation besides the specification of inputs and outputs. Specifically, there are no limits on the amount of data that can be requested from the provider. This enables algorithm maintainers to implement algorithms in non-scalable ways, such as re-training the detection model on long time ranges before each evaluation of the detection model. It also doesn't prevent the system (and its data sources) from mista [...]
+
+Another limitation here is the restriction of parallelism on a per-flow basis. Pipelines and algorithms can contain internal state during execution which is not stored in any external meta data store. This enables algorithm developers to create arbitrary logic, but restricts parallelism to a single serial thread of execution per flow in order to avoid the complexity of synchronization and distributed processing.
+
+"nesting and non-nesting configuration" vs "implicit coupling via property key injection"
+**********************************************************************************************
+There is a fundamental trade-off between separately configuring individual metric- or dimension-level alerts and setting up a single detector with overrides specific to a single sub-task of detection. Furthermore, this configuration may be injected from a wrapper pipeline down into a nested pipeline. We explicitly chose to use a single, all-encompassing configuration per detection use-case to allow consistent handling of related anomalies in a single flow, for example for merging or clus [...]
+
+"generalized configuration object" vs "static type safety of config"
+****************************************************************************
+Configuration of pipelines could be served as statically defined config classes or semi-structured (and dynamically typed) key-value maps. Static objects provide type-safety and would allow static checking of configuration correctness. They add overhead for pipeline development and code however. The alternative delays error checking to runtime, i.e. only when the configured pipeline is instantiated and executed. This approach is more lightweight and flexible in terms of development. When [...]
+
+"atomic execution" vs "redundant computation"
+*************************************************
+Anomaly detection isn't a purely online process, i.e. detection sometimes changes its decisions about the past state of the world after detection already on this past time range. For example, a new but short short outlier may be ignored by detection initially, but may be re-classified as an anomaly when the following data points are generated and show similar anomalous behavior. ThirdEye's legacy detection pipeline chose to store both "candidates" and "confirmed" anomalies in the data ba [...]
+
+"serial execution, custom and re-usable wrappers" vs "parallel execution pipeline parts"
+*********************************************************************************************
+Parallelism in ThirdEye performs on job level, but not per task. This allows users to specify arbitrary flows of exploration, detection, merging, and grouping tasks as all the state is available in a single place during execution (see atomic execution above). The trade-off here comes from a limit to scaling of extremely large singular detection flows that cannot execute serially. This can be mitigated by splitting the job into multiple independent ones, effectively allowing the user to c [...]
+
+
diff --git a/thirdeye/docs/detection_pipeline_execution_flow.rst b/thirdeye/docs/detection_pipeline_execution_flow.rst
new file mode 100644
index 0000000..d08eac9
--- /dev/null
+++ b/thirdeye/docs/detection_pipeline_execution_flow.rst
@@ -0,0 +1,192 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+.. _detection-pipeline-execution-flow:
+
+######################################
+Detection Pipeline Execution Flow
+######################################
+
+Background & Motivation
+###########################
+Current detection pipeline does not support business rules well. The user-base of ThirdEye is growing constantly. A lot of anomaly detection use-case comes with an additional set of business rules. For example, growth team want to filter out the anomalies whose site-wide impact is less than a certain threshold. They need to group anomalies across different dimensions. Some teams want to set up anomaly detection using threshold rules or want to use a fixed list of sub-dimensions to monito [...]
+
+Users also have no way to configure their business rule without reaching out to the pipeline maintainer to change the config JSON manually, which is not scalable. Some users, although has their specific inclusion/exclusion rules, still want to utilize the auto-tuned algorithm-based detection pipeline. This is not achievable in the current pipeline.
+
+Due to the limitations described above, we introduce the composite pipeline flow in the new detection pipeline framework to achieve the following goals:
+
+ - More flexibility of adding user-defined business rules into the pipeline
+
+ - User-friendly configuration of the detection rules
+
+ - Robustness and testability
+
+Design & Implementation
+##########################
+
+Composite pipeline flow
+*************************
+The pipeline is shown as follows.
+
+.. image:: https://user-images.githubusercontent.com/4448437/88265403-48f43480-cc82-11ea-9efe-1c30016a6669.png
+ :width: 500
+
+Dimension Exploration:
+========================
+Dimension drill down for user-defined dimensions. For example, explore county dimension where continent = Europe.
+
+Dimension Filter:
+=====================
+Filter out dimensions based on some business logic criteria. For example, only explore the dimension whose contribution to overall metric > 5%.
+
+Rule Detection:
+==================
+User specified rule for anomaly detection. For example, if percentage change WoW > 10%, fires an anomaly.
+
+Algorithm Detection:
+=====================
+Existing anomaly detection algorithms. Such as sign test, spline regression, etc.
+
+Algorithm alert filter:
+========================
+Existing auto-tune alert filters.
+
+Merger:
+=========
+For each dimension, merge anomalies based on time. See more detailed discussion about merging logic below.
+
+Rule filter:
+==============
+Exclusion filter for anomalies defined by user to filter out the anomalies they don’t want to receive. For example, if within the anomaly time range, the site wide impact of this metric in this dimension is less than 5% , don't classify this as an anomaly .
+
+Grouper:
+==========
+Groups anomalies across different dimensions.
+
+The algorithm detection and alert filter will provide backward-compatibility to existing anomaly function interface.
+
+For each stage, we provides interfaces so that they can be pluggable. User can provide any kind of business logic to customized the logic of each stage. The details of the interfaces are listed in this page: Detection pipeline interfaces.
+
+Pros of this pipeline:
+========================
+* Users can defines inclusion rules to detect anomalies.
+* Users won't receive the anomalies they explicitly filtered out if they set up the exclusion rule-based filter.
+* For each dimension, users won't see duplicated anomalies generated by algorithm & rule pipeline for any time range, since they are merged based on time.
+
+
+
+Alternative Pipeline flow designs:
+*************************************
+1.
+
+.. image:: https://user-images.githubusercontent.com/4448437/88265408-4b568e80-cc82-11ea-83e7-a833663a68ed.png
+ :width: 500
+
+Pros of this pipeline:
+========================
+* Users can defines inclusion rules to detect anomalies.
+* Users won't receive the anomalies they explicitly filtered out if they set up the exclusion rule-based filter.
+* Users won't see duplicated anomalies generated by algorithm & rule pipeline, since they are merged based on time.
+
+Cons of this pipeline:
+========================
+* The algorithm alert filter might filter out the anomalies generated by user specified rules, i.e. users could miss anomalies they want to see.
+
+
+2.
+
+.. image:: https://user-images.githubusercontent.com/4448437/88265411-4e517f00-cc82-11ea-947a-04bee30ca08c.png
+ :width: 500
+
+Pros of this pipeline:
+========================
+* Users can defines inclusion rules to detect anomalies.
+* Users won't see duplicated anomalies generated by algorithm & rule pipeline, since they are merged based on time.
+
+Cons of this pipeline:
+========================
+* Users will still see the anomaly they set rules to explicitly filter out. Because the anomalies generated by algorithm detection pipeline does not filtered by user’s exclusion rule.
+
+As discussed above, we recommend to use the first discussed design as default. The detection framework itself still has the flexibility of executing different type of flows if this is needed later.
+
+
+
+Merging logic
+#################
+Merging happens either when merging anomalies within a rule/algorithm detection flow or merge anomalies generated by different flows. Merger's behavior is slightly different in these two conditions.
+
+Merging only rule-detected anomalies or rule-detected anomalies
+********************************************************************
+Do time-based merging only. Do not keep anomalies before merging.
+
+Merging both rule-detected anomalies and algorithm-detected anomalies
+**********************************************************************
+There will be 3 cases when merging two anomalies:
+
+.. image:: https://user-images.githubusercontent.com/4448437/88265414-501b4280-cc82-11ea-904e-83fd54e3a157.png
+ :width: 500
+
+Solution to case 2:
+=====================
+1. Merge all time intervals in both anomalies.
+-------------------------------------------------
+In this example, will send A-D as the anomaly.
+
+Pros:
+======
+* Users will not receive duplicated anomaly for any specific range.
+* Improves the recall.
+
+Cons:
+======
+* Users will receive an extended anomaly range. More period to investigate
+
+
+2. Only classify as an anomaly for the overlapped interval.
+-------------------------------------------------------------
+In this example, will send C-B as the anomaly.
+
+Pros:
+======
+* User will not receive duplicated anomaly for any specific range.
+* Improved the precision. The anomaly range is shortened. User has less period to investigate.
+
+Cons:
+======
+* User could miss the anomaly period he explicitly set rule to detect. Because the merger might chop off the anomaly period. Reduce the recall.
+
+
+3. Don’t merge, send two anomalies.
+-----------------------------------------
+In this example, will send A-B and C-D as two anomalies.
+
+Pros:
+======
+* Improves the recall
+
+Cons:
+======
+* User will receive duplicated anomaly for a specific time range, in this example for C-B.
+* User has more workload to investigate because of more anomalies
+
+
+
+As discuss above, we set merger to behave like solution 1 by default, i.e, merger merges the time period. The merger will keep the anomalies before merging as the child anomalies. This allows tracing back to the anomalies generated by different algorithms/rules.
+
+
diff --git a/thirdeye/docs/index.rst b/thirdeye/docs/index.rst
index d17415a..175bfce 100644
--- a/thirdeye/docs/index.rst
+++ b/thirdeye/docs/index.rst
@@ -29,3 +29,5 @@ ThirdEye
datasources
caches
alert_setup
+ thirdeye_architecture
+ thirdeye_ui_mocks
diff --git a/thirdeye/docs/alert_setup.rst b/thirdeye/docs/thirdeye_architecture.rst
similarity index 84%
copy from thirdeye/docs/alert_setup.rst
copy to thirdeye/docs/thirdeye_architecture.rst
index bda73f0..3232c52 100644
--- a/thirdeye/docs/alert_setup.rst
+++ b/thirdeye/docs/thirdeye_architecture.rst
@@ -17,16 +17,13 @@
.. under the License.
..
-.. _alert-setup:
+.. _thirdeye-architecture:
-Alert Setup
-############
+ThirdEye Architecture
+##########################
.. toctree::
:maxdepth: 1
- basic_config
- advanced_config
- templates
- appendix
- contribute_detection
\ No newline at end of file
+ detection_pipeline_architecture
+ detection_pipeline_execution_flow
diff --git a/thirdeye/docs/alert_setup.rst b/thirdeye/docs/thirdeye_ui_mocks.rst
similarity index 58%
copy from thirdeye/docs/alert_setup.rst
copy to thirdeye/docs/thirdeye_ui_mocks.rst
index bda73f0..7a1cd1a 100644
--- a/thirdeye/docs/alert_setup.rst
+++ b/thirdeye/docs/thirdeye_ui_mocks.rst
@@ -17,16 +17,15 @@
.. under the License.
..
-.. _alert-setup:
+.. _thirdeye-ui-mocks:
-Alert Setup
-############
+ThirdEye UI Mocks
+##########################
.. toctree::
:maxdepth: 1
- basic_config
- advanced_config
- templates
- appendix
- contribute_detection
\ No newline at end of file
+ :download: `Entity (multi-metrics) Monitoring <https://github.com/apache/incubator-pinot/files/4964811/ThirdEye_EntityMonitoring.pdf>`
+ :download: `Subscription group Management <https://github.com/apache/incubator-pinot/files/4964812/ThirdEye_SubscriptionGroups_Management.pdf>`
+ :download: `Suppress Alert <https://github.com/apache/incubator-pinot/files/4964813/ThirdEye_Suppress_Alerts.pdf>`
+ :download: `SLA monitoring <https://github.com/apache/incubator-pinot/files/4964810/ThirdEye_SLA_monitoring.pdf>`
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org