You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xh...@apache.org on 2019/07/12 00:41:34 UTC
[incubator-pinot] 01/01: [TE] added ThirdEye configuration documents
This is an automated email from the ASF dual-hosted git repository.
xhsun pushed a commit to branch document_configuration
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 6b833b95d7bd6aea3b8ec0b3c6ed86e4a29c2d13
Author: Xiaohui Sun <xh...@xhsun-mn3.linkedin.biz>
AuthorDate: Thu Jul 11 17:41:18 2019 -0700
[TE] added ThirdEye configuration documents
---
thirdeye/docs/configuration.rst | 195 ++++++++++++++++++++++++++++++++++++++++
thirdeye/docs/index.rst | 2 +-
thirdeye/docs/production.rst | 27 ------
3 files changed, 196 insertions(+), 28 deletions(-)
diff --git a/thirdeye/docs/configuration.rst b/thirdeye/docs/configuration.rst
new file mode 100644
index 0000000..4a78969
--- /dev/null
+++ b/thirdeye/docs/configuration.rst
@@ -0,0 +1,195 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Configuration Settings
+=======================
+
+Overview
+-------------
+ThirdEye could be deployed on single machine or deployed in clusters.
+
+- Dashboard servers are used to host web applications.
+The ``org.apache.pinot.thirdeye.dashboard.ThirdEyeDashboardApplication`` class is the entry point.
+
+``dashboard.yml`` and ``rca.yml`` are used to configure dashboard servers.
+
+- Backend servers are used to schedule tasks or run the tasks.
+The ``org.apache.pinot.thirdeye.anomaly.ThirdEyeAnomalyApplication`` class is the entry point.
+
+``detector.yml``, ``persistence.yml`` and ``data-sources-config.yml`` are used to configure backend servers.
+
+.. image:: https://user-images.githubusercontent.com/44730481/61093367-fd684300-a3fe-11e9-943e-d20ba9651528.png
+
+
+detector.yml
+^^^^^^^^^^^^^^^
+
+ThirdEye uses this file to configure the backend server.
+
+You can deploy ThirdEye to multiple nodes with specific modules enabled in each node.
+
+Here are a list of modules you can configure in this file:
+
+- autoload: Load Pinot metrics metadata automatically.
+- holidayEventsLoader: Load holiday events from Google Calendar.
+- monitor: Used to do clean up tasks. By default ThirdEye will delete - detection tasks that are older than 30 days and alert tasks that are older than 10 days.
+- worker: Handles the actual tasks to do anomaly detection or alerting. You can deploy multiple workers to share the load.
+- detectionPipeline: Scheduler to generate detection tasks.
+- detectionAlert: Scheduler to generate alert tasks.
+
+.. image:: https://user-images.githubusercontent.com/44730481/61093449-6d76c900-a3ff-11e9-85e6-01201ea71eeb.png
+
+To enable one module, you can change the module's value to "true".
+
+For example, below configures a node with worker and scheduler enabled.
+
+.. code-block:: yaml
+
+ holidayEventsLoader: false
+ monitor: false
+ pinotProxy: false
+ worker: true
+ detectionPipeline: true
+ detectionAlert: true
+
+To have the minimum system running you need to enable "worker", "monitor", "detectionPipeline" and "detectionAlert".
+
+Besides the module configuration you can configure the other followings in this file:
+
+- SMTP configuration: Configure SMTP server which is used to send alert mail.
+- Log configuration: SLF4J configurations.
+- Server ports: Endpoint ports for backend servers.
+- Swagger configuration.
+- PhantomJSPath: PhantomJS is used to generate anomaly metrics screenshots which are attached in alert mail.
+
+
+persistence.yml
+^^^^^^^^^^^^^^^^^^^
+
+ThirdEye uses MySQL to store all the metadata. This file is used to configure MySQL database instance.
+
+databaseConfiguration: url: user: password: driver: com.mysql.jdbc.Driver
+Here is an example:
+
+databaseConfiguration: url: jdbc:mysql:///thirdeye?autoReconnect=true user: te_dev password: xxxxx driver: com.mysql.jdbc.Driver
+
+
+data-sources-config.yml
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+ThirdEye doesn't store the actual metrics but will pull the metrics using data source loaders. This file controls the metrics data sources.
+
+Here is an example used in ThirdEye production which connects to two data sources: PinotThirdEyeDataSource and SqlThirdEyeDataSource.
+
+Please note ThirdEye support MySQL data source, and this configuration is different with persistence.yml.
+
+.. code-block:: yaml
+
+ dataSourceConfigs:
+ - className: org.apache.pinot.thirdeye.datasource.pinot.PinotThirdEyeDataSource
+ properties:
+ zookeeperUrl: '<zookeeperurl>'
+ clusterName: '<clustername>'
+ controllerConnectionScheme: 'https'
+ controllerHost: '<hostname>'
+ controllerPort: <port>
+ cacheLoaderClassName: org.apache.pinot.thirdeye.datasource.pinot.PinotD2ResponseCacheLoader
+ metadataSourceConfigs:
+ - className: org.apache.pinot.thirdeye.auto.onboard.AutoOnboardPinotMetadataSource
+ - className: org.apache.pinot.thirdeye.datasource.sql.SqlThirdEyeDataSource
+ properties:
+ MySQL:
+ - db:
+ te: 'jdbc:mysql://<mysqlurl>/thirdeye?autoReconnect=true'
+ user: 'thirdeye'
+ password: '<password>'
+
+dashboard.yml
+^^^^^^^^^^^^^^
+
+Controls settings relate to web application servers. The followings are configured here:
+
+- LDAP authentication. To enable LDAP authentication, change "authEnabled" to "true".
+
+.. code-block:: yaml
+
+ authConfig:
+ authEnabled: true
+ authKey: <authentication_key>
+ ldapUrl: <ldap_url>
+ domainSuffix:
+ - linkedin.biz
+ cacheTTL: 3600
+ cookieTTL: 604800
+ adminUsers:
+ - user1
+ - user2
+
+- Root cause analysis (RCA) configuration: Control thread pool size for RCA pipelines. Default is 5.
+- Dashboard host and endpoints configuration.
+- Swagger configuration.
+
+
+rca.yml
+^^^^^^^^
+This configures the RCA pipelines, which is used to either do metrics analysis or loads events from different systems.
+
+These pipelines are called online and not pre-loaded.
+
+Each pipeline derives from org.apache.pinot.thirdeye.rootcause.Pipeline class, and has "inputNames", "outputName", "className" and "properties". One pipeline can take another pipeline's output as input and it is a DAG.
+
+The "className" is used to create instances using reflection.
+
+
+FAQ
+^^^^
+
+How to add a new application?
+-------------------------------
+
+Application is a group of users. ThirdEye can show anomalies grouped by applications.
+
+There are two ways to add a new application
+
+Send a post message
+---------------------
+
+curl -vX POST '.../thirdeye/entity?entityType=APPLICATION' -H "Content-Type: application/json" -H 'Cookie: te_auth=[REPLACE_TOKEN_HERE]' --data-binary @application.json
+
+Example payload could be:
+
+.. code-block:: json
+
+ {
+ "application": "replace-application-name",
+ "recipients": "owner-of-this-application@company.com"
+ }
+
+From admin page
+----------------
+
+1. Navigate to admin page: http://localhost:1426/thirdeye-admin
+2. Select an existing application from "Entity Editor".
+
+.. image:: https://user-images.githubusercontent.com/44730481/61093646-61d7d200-a400-11e9-8517-0b46bd33fe2a.png
+
+3. Put the application and recipients in the editor, and leave id, version etc unspecified.
+.. image:: https://user-images.githubusercontent.com/44730481/61093659-6c926700-a400-11e9-8690-6a1742671e5e.png
+
+4. Click "load to editor" then click "submit".
diff --git a/thirdeye/docs/index.rst b/thirdeye/docs/index.rst
index 313a6a3..0446113 100644
--- a/thirdeye/docs/index.rst
+++ b/thirdeye/docs/index.rst
@@ -25,6 +25,6 @@ ThirdEye
:maxdepth: 2
introduction
+ configuration
datasources
- production
alert_setup
\ No newline at end of file
diff --git a/thirdeye/docs/production.rst b/thirdeye/docs/production.rst
deleted file mode 100644
index cecbb91..0000000
--- a/thirdeye/docs/production.rst
+++ /dev/null
@@ -1,27 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. _production:
-
-Production Settings
-====================
-
-ThirdEye relies on a central meta data store to coordinate its workers and frontend processes. The first step towards moving ThirdEye into production should therefore be the setup of a dedicated (MySQL) database instance. You can use the ``thirdeye-pinot/src/main/resources/schema/create-schema.sql`` script to create your tables. Then, update the ``thirdeye-pinot/config/persistence.yml`` file with path and credentials. Once you have a dedicated database instance, you can run backend and f [...]
-
-The next step could be the configuration of the holiday auto-loader. The holiday auto loader connects to the Google Calendar API. Once you obtain an API token, place it in ``thirdeye-pinot/config/holiday-loader-key.json`` and in ``thirdeye-pinot/config/detector.yml`` set holidayEventsLoader: true. Once the backend worker is restarted, it will periodically update the local cache of holiday events for ThirdEye's detection and Root-Cause Analysis components.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org