You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by xh...@apache.org on 2019/07/12 00:41:33 UTC
[incubator-pinot] branch document_configuration created (now
6b833b9)
This is an automated email from the ASF dual-hosted git repository.
xhsun pushed a change to branch document_configuration
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
at 6b833b9 [TE] added ThirdEye configuration documents
This branch includes the following new commits:
new 6b833b9 [TE] added ThirdEye configuration documents
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[incubator-pinot] 01/01: [TE] added ThirdEye configuration documents
Posted by xh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xhsun pushed a commit to branch document_configuration
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 6b833b95d7bd6aea3b8ec0b3c6ed86e4a29c2d13
Author: Xiaohui Sun <xh...@xhsun-mn3.linkedin.biz>
AuthorDate: Thu Jul 11 17:41:18 2019 -0700
[TE] added ThirdEye configuration documents
---
thirdeye/docs/configuration.rst | 195 ++++++++++++++++++++++++++++++++++++++++
thirdeye/docs/index.rst | 2 +-
thirdeye/docs/production.rst | 27 ------
3 files changed, 196 insertions(+), 28 deletions(-)
diff --git a/thirdeye/docs/configuration.rst b/thirdeye/docs/configuration.rst
new file mode 100644
index 0000000..4a78969
--- /dev/null
+++ b/thirdeye/docs/configuration.rst
@@ -0,0 +1,195 @@
+..
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+..
+
+Configuration Settings
+=======================
+
+Overview
+-------------
+ThirdEye could be deployed on single machine or deployed in clusters.
+
+- Dashboard servers are used to host web applications.
+The ``org.apache.pinot.thirdeye.dashboard.ThirdEyeDashboardApplication`` class is the entry point.
+
+``dashboard.yml`` and ``rca.yml`` are used to configure dashboard servers.
+
+- Backend servers are used to schedule tasks or run the tasks.
+The ``org.apache.pinot.thirdeye.anomaly.ThirdEyeAnomalyApplication`` class is the entry point.
+
+``detector.yml``, ``persistence.yml`` and ``data-sources-config.yml`` are used to configure backend servers.
+
+.. image:: https://user-images.githubusercontent.com/44730481/61093367-fd684300-a3fe-11e9-943e-d20ba9651528.png
+
+
+detector.yml
+^^^^^^^^^^^^^^^
+
+ThirdEye uses this file to configure the backend server.
+
+You can deploy ThirdEye to multiple nodes with specific modules enabled in each node.
+
+Here are a list of modules you can configure in this file:
+
+- autoload: Load Pinot metrics metadata automatically.
+- holidayEventsLoader: Load holiday events from Google Calendar.
+- monitor: Used to do clean up tasks. By default ThirdEye will delete - detection tasks that are older than 30 days and alert tasks that are older than 10 days.
+- worker: Handles the actual tasks to do anomaly detection or alerting. You can deploy multiple workers to share the load.
+- detectionPipeline: Scheduler to generate detection tasks.
+- detectionAlert: Scheduler to generate alert tasks.
+
+.. image:: https://user-images.githubusercontent.com/44730481/61093449-6d76c900-a3ff-11e9-85e6-01201ea71eeb.png
+
+To enable one module, you can change the module's value to "true".
+
+For example, below configures a node with worker and scheduler enabled.
+
+.. code-block:: yaml
+
+ holidayEventsLoader: false
+ monitor: false
+ pinotProxy: false
+ worker: true
+ detectionPipeline: true
+ detectionAlert: true
+
+To have the minimum system running you need to enable "worker", "monitor", "detectionPipeline" and "detectionAlert".
+
+Besides the module configuration you can configure the other followings in this file:
+
+- SMTP configuration: Configure SMTP server which is used to send alert mail.
+- Log configuration: SLF4J configurations.
+- Server ports: Endpoint ports for backend servers.
+- Swagger configuration.
+- PhantomJSPath: PhantomJS is used to generate anomaly metrics screenshots which are attached in alert mail.
+
+
+persistence.yml
+^^^^^^^^^^^^^^^^^^^
+
+ThirdEye uses MySQL to store all the metadata. This file is used to configure MySQL database instance.
+
+databaseConfiguration: url: user: password: driver: com.mysql.jdbc.Driver
+Here is an example:
+
+databaseConfiguration: url: jdbc:mysql:///thirdeye?autoReconnect=true user: te_dev password: xxxxx driver: com.mysql.jdbc.Driver
+
+
+data-sources-config.yml
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+ThirdEye doesn't store the actual metrics but will pull the metrics using data source loaders. This file controls the metrics data sources.
+
+Here is an example used in ThirdEye production which connects to two data sources: PinotThirdEyeDataSource and SqlThirdEyeDataSource.
+
+Please note ThirdEye support MySQL data source, and this configuration is different with persistence.yml.
+
+.. code-block:: yaml
+
+ dataSourceConfigs:
+ - className: org.apache.pinot.thirdeye.datasource.pinot.PinotThirdEyeDataSource
+ properties:
+ zookeeperUrl: '<zookeeperurl>'
+ clusterName: '<clustername>'
+ controllerConnectionScheme: 'https'
+ controllerHost: '<hostname>'
+ controllerPort: <port>
+ cacheLoaderClassName: org.apache.pinot.thirdeye.datasource.pinot.PinotD2ResponseCacheLoader
+ metadataSourceConfigs:
+ - className: org.apache.pinot.thirdeye.auto.onboard.AutoOnboardPinotMetadataSource
+ - className: org.apache.pinot.thirdeye.datasource.sql.SqlThirdEyeDataSource
+ properties:
+ MySQL:
+ - db:
+ te: 'jdbc:mysql://<mysqlurl>/thirdeye?autoReconnect=true'
+ user: 'thirdeye'
+ password: '<password>'
+
+dashboard.yml
+^^^^^^^^^^^^^^
+
+Controls settings relate to web application servers. The followings are configured here:
+
+- LDAP authentication. To enable LDAP authentication, change "authEnabled" to "true".
+
+.. code-block:: yaml
+
+ authConfig:
+ authEnabled: true
+ authKey: <authentication_key>
+ ldapUrl: <ldap_url>
+ domainSuffix:
+ - linkedin.biz
+ cacheTTL: 3600
+ cookieTTL: 604800
+ adminUsers:
+ - user1
+ - user2
+
+- Root cause analysis (RCA) configuration: Control thread pool size for RCA pipelines. Default is 5.
+- Dashboard host and endpoints configuration.
+- Swagger configuration.
+
+
+rca.yml
+^^^^^^^^
+This configures the RCA pipelines, which is used to either do metrics analysis or loads events from different systems.
+
+These pipelines are called online and not pre-loaded.
+
+Each pipeline derives from org.apache.pinot.thirdeye.rootcause.Pipeline class, and has "inputNames", "outputName", "className" and "properties". One pipeline can take another pipeline's output as input and it is a DAG.
+
+The "className" is used to create instances using reflection.
+
+
+FAQ
+^^^^
+
+How to add a new application?
+-------------------------------
+
+Application is a group of users. ThirdEye can show anomalies grouped by applications.
+
+There are two ways to add a new application
+
+Send a post message
+---------------------
+
+curl -vX POST '.../thirdeye/entity?entityType=APPLICATION' -H "Content-Type: application/json" -H 'Cookie: te_auth=[REPLACE_TOKEN_HERE]' --data-binary @application.json
+
+Example payload could be:
+
+.. code-block:: json
+
+ {
+ "application": "replace-application-name",
+ "recipients": "owner-of-this-application@company.com"
+ }
+
+From admin page
+----------------
+
+1. Navigate to admin page: http://localhost:1426/thirdeye-admin
+2. Select an existing application from "Entity Editor".
+
+.. image:: https://user-images.githubusercontent.com/44730481/61093646-61d7d200-a400-11e9-8517-0b46bd33fe2a.png
+
+3. Put the application and recipients in the editor, and leave id, version etc unspecified.
+.. image:: https://user-images.githubusercontent.com/44730481/61093659-6c926700-a400-11e9-8690-6a1742671e5e.png
+
+4. Click "load to editor" then click "submit".
diff --git a/thirdeye/docs/index.rst b/thirdeye/docs/index.rst
index 313a6a3..0446113 100644
--- a/thirdeye/docs/index.rst
+++ b/thirdeye/docs/index.rst
@@ -25,6 +25,6 @@ ThirdEye
:maxdepth: 2
introduction
+ configuration
datasources
- production
alert_setup
\ No newline at end of file
diff --git a/thirdeye/docs/production.rst b/thirdeye/docs/production.rst
deleted file mode 100644
index cecbb91..0000000
--- a/thirdeye/docs/production.rst
+++ /dev/null
@@ -1,27 +0,0 @@
-..
-.. Licensed to the Apache Software Foundation (ASF) under one
-.. or more contributor license agreements. See the NOTICE file
-.. distributed with this work for additional information
-.. regarding copyright ownership. The ASF licenses this file
-.. to you under the Apache License, Version 2.0 (the
-.. "License"); you may not use this file except in compliance
-.. with the License. You may obtain a copy of the License at
-..
-.. http://www.apache.org/licenses/LICENSE-2.0
-..
-.. Unless required by applicable law or agreed to in writing,
-.. software distributed under the License is distributed on an
-.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-.. KIND, either express or implied. See the License for the
-.. specific language governing permissions and limitations
-.. under the License.
-..
-
-.. _production:
-
-Production Settings
-====================
-
-ThirdEye relies on a central meta data store to coordinate its workers and frontend processes. The first step towards moving ThirdEye into production should therefore be the setup of a dedicated (MySQL) database instance. You can use the ``thirdeye-pinot/src/main/resources/schema/create-schema.sql`` script to create your tables. Then, update the ``thirdeye-pinot/config/persistence.yml`` file with path and credentials. Once you have a dedicated database instance, you can run backend and f [...]
-
-The next step could be the configuration of the holiday auto-loader. The holiday auto loader connects to the Google Calendar API. Once you obtain an API token, place it in ``thirdeye-pinot/config/holiday-loader-key.json`` and in ``thirdeye-pinot/config/detector.yml`` set holidayEventsLoader: true. Once the backend worker is restarted, it will periodically update the local cache of holiday events for ThirdEye's detection and Root-Cause Analysis components.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org