[GitHub] [skywalking] dmsolr commented on pull request #7039: WIP: Support hot update of MAL

[GitBox <gi...@apache.org>]

dmsolr commented on pull request #7039:
URL: https://github.com/apache/skywalking/pull/7039#issuecomment-869462815


   
   SkyWalking Metrics data comes from MeterSystem and OALEngine. The source of `MeterSystem` includes Receiver and Fetcher. 
   
   > we had many receivers includes Zabbix, OpenTelmetry, JVM, and Meter. ..
   
   Currently, we have had the Prometheus Fetcher which scraps metrics sampling from the Prometheus exporter. It could convert Prometheus Metrics to MeterSystem's Sample by MAL rules.
   
   ![image](https://user-images.githubusercontent.com/29735230/123601098-dabcd800-d829-11eb-8516-b3a2ebd7e0fe.png)
   
   
   There are 3 types of operators, ADD, UPDATE and DELETE.
   1. For ADD a new metrics, not conflict
   2. For DELETE, there are 2 options, 1. None operator, 2. Always drop table.
   3. For UPDATE, For now, we provided 3 options(Option c as default).
     1. Always keep old data(table)
     2. Always discard old data(table)
     3. Interrupted when table updated
   
   Above description is not suitation for ElasticSearch. In RMDBS(mysql/postgresql), tidb and influxdb. A metrics represent a model, and a table(measuremnt in influxdb).  And the table name is the mode name. So, for update, 
   1. Rename table name after a metrics rule udpated for keeping old data in DB.
   2. Drop the table before creating a new metrics writing for discard old data.
   3. None operation without the table structure changed. Otherwise, an exception is thrown for interrupting the OAP.
   
   In ElasticSearch(after #6499), have many difference with other storage plugins. SW has put all metrics into one index. And identified metrics data by the function name and mode name. 
   
   1. For ADD a new metrics, not conflict also.
   2. For DELETE a metrics, 1. Waiting for TTL to clean up. 2.  Execution a query for delete.
   3. For UPDATE, 
     1. Waiting for TTL to sweep.
     2. Execution delete statement.
   
   In ElasticSearch case, there is no error cause by MAL updated. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org