You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/02/16 08:29:02 UTC

[GitHub] [skywalking] wu-sheng opened a new pull request #4368: Support Daily Index Step feature in the OAP

wu-sheng opened a new pull request #4368: Support Daily Index Step feature in the OAP
URL: https://github.com/apache/skywalking/pull/4368

### Daily Index Step
Daily index step(`storage/elasticsearch/dayStep`, default 1) represents the index creation period. In this period, several days(dayStep value)' metrics are saved.

Mostly, users don't need to change the value manually. As SkyWalking is designed to observe large scale distributed system.
But in some specific cases, users want to set a long TTL value, such as more than 60 days, but their ElasticSearch cluster isn't powerful due to the low traffic in the production environment.
This value could be increased to 5(or more), if users could make sure single one index could support these days(5 in this case) metrics and traces.

Such as, if dayStep == 11,
1. data in [2000-01-01, 2000-01-11] will be merged into the index-20000101.
1. data in [2000-01-12, 2000-01-22] will be merged into the index-20000112.

NOTICE, TTL deletion would be affected by these. You should set an extra more dayStep in your TTL. Such as you want to TTL == 30 days and dayStep == 10, you actually need to set TTL = 40;

### Data in the storage
The data in the dayStep period are merged into one index.
![image](https://user-images.githubusercontent.com/5441976/74601432-5c9ade00-50d9-11ea-8e9a-93455cd26cee.png)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [skywalking] JaredTan95 merged pull request #4368: Support Daily Index Step feature in the OAP

Posted by GitBox <gi...@apache.org>.

JaredTan95 merged pull request #4368: Support Daily Index Step feature in the OAP
URL: https://github.com/apache/skywalking/pull/4368
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [skywalking] wu-sheng commented on issue #4368: Support Daily Index Step feature in the OAP

Posted by GitBox <gi...@apache.org>.

wu-sheng commented on issue #4368: Support Daily Index Step feature in the OAP
URL: https://github.com/apache/skywalking/pull/4368#issuecomment-586702690
 
 
   @JaredTan95 Yes, so if the user only have low traffic, they could have 70-80 indexes but hosting 1-2weeks later with a small ES cluster.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [skywalking] codecov-io commented on issue #4368: Support Daily Index Step feature in the OAP

Posted by GitBox <gi...@apache.org>.

codecov-io commented on issue #4368: Support Daily Index Step feature in the OAP
URL: https://github.com/apache/skywalking/pull/4368#issuecomment-586686016
 
 
   # [Codecov](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=h1) Report
   > Merging [#4368](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=desc) into [master](https://codecov.io/gh/apache/skywalking/commit/329f7e150b9cb25624d627497a7a8c620cf92f68?src=pr&el=desc) will **decrease** coverage by `0.26%`.
   > The diff coverage is `18.86%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/skywalking/pull/4368/graphs/tree.svg?width=650&token=qrILxY5yA8&height=150&src=pr)](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #4368      +/-   ##
   ==========================================
   - Coverage   26.35%   26.09%   -0.27%     
   ==========================================
     Files        1182     1182              
     Lines       26975    27007      +32     
     Branches     3719     3723       +4     
   ==========================================
   - Hits         7110     7047      -63     
   - Misses      19232    19324      +92     
   - Partials      633      636       +3
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [...sticsearch/StorageModuleElasticsearchProvider.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItc3RvcmFnZS1wbHVnaW4vc3RvcmFnZS1lbGFzdGljc2VhcmNoLXBsdWdpbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2t5d2Fsa2luZy9vYXAvc2VydmVyL3N0b3JhZ2UvcGx1Z2luL2VsYXN0aWNzZWFyY2gvU3RvcmFnZU1vZHVsZUVsYXN0aWNzZWFyY2hQcm92aWRlci5qYXZh) | `0% <0%> (ø)` | :arrow_up: |
   | [.../plugin/elasticsearch/base/HistoryDeleteEsDAO.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItc3RvcmFnZS1wbHVnaW4vc3RvcmFnZS1lbGFzdGljc2VhcmNoLXBsdWdpbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2t5d2Fsa2luZy9vYXAvc2VydmVyL3N0b3JhZ2UvcGx1Z2luL2VsYXN0aWNzZWFyY2gvYmFzZS9IaXN0b3J5RGVsZXRlRXNEQU8uamF2YQ==) | `0% <0%> (ø)` | :arrow_up: |
   | [.../plugin/elasticsearch/base/StorageEsInstaller.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItc3RvcmFnZS1wbHVnaW4vc3RvcmFnZS1lbGFzdGljc2VhcmNoLXBsdWdpbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2t5d2Fsa2luZy9vYXAvc2VydmVyL3N0b3JhZ2UvcGx1Z2luL2VsYXN0aWNzZWFyY2gvYmFzZS9TdG9yYWdlRXNJbnN0YWxsZXIuamF2YQ==) | `0% <0%> (ø)` | :arrow_up: |
   | [.../server/core/alarm/provider/grpc/GRPCCallback.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItYWxhcm0tcGx1Z2luL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9za3l3YWxraW5nL29hcC9zZXJ2ZXIvY29yZS9hbGFybS9wcm92aWRlci9ncnBjL0dSUENDYWxsYmFjay5qYXZh) | `59.42% <0%> (-1.45%)` | :arrow_down: |
   | [...walking/oap/server/core/analysis/Downsampling.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2t5d2Fsa2luZy9vYXAvc2VydmVyL2NvcmUvYW5hbHlzaXMvRG93bnNhbXBsaW5nLmphdmE=) | `0% <0%> (ø)` | :arrow_up: |
   | [...lasticsearch/StorageModuleElasticsearchConfig.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItc3RvcmFnZS1wbHVnaW4vc3RvcmFnZS1lbGFzdGljc2VhcmNoLXBsdWdpbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2t5d2Fsa2luZy9vYXAvc2VydmVyL3N0b3JhZ2UvcGx1Z2luL2VsYXN0aWNzZWFyY2gvU3RvcmFnZU1vZHVsZUVsYXN0aWNzZWFyY2hDb25maWcuamF2YQ==) | `0% <0%> (ø)` | :arrow_up: |
   | [...icsearch7/StorageModuleElasticsearch7Provider.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItc3RvcmFnZS1wbHVnaW4vc3RvcmFnZS1lbGFzdGljc2VhcmNoNy1wbHVnaW4vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NreXdhbGtpbmcvb2FwL3NlcnZlci9zdG9yYWdlL3BsdWdpbi9lbGFzdGljc2VhcmNoNy9TdG9yYWdlTW9kdWxlRWxhc3RpY3NlYXJjaDdQcm92aWRlci5qYXZh) | `0% <0%> (ø)` | :arrow_up: |
   | [...age/plugin/elasticsearch/base/TimeSeriesUtils.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9zZXJ2ZXItc3RvcmFnZS1wbHVnaW4vc3RvcmFnZS1lbGFzdGljc2VhcmNoLXBsdWdpbi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2t5d2Fsa2luZy9vYXAvc2VydmVyL3N0b3JhZ2UvcGx1Z2luL2VsYXN0aWNzZWFyY2gvYmFzZS9UaW1lU2VyaWVzVXRpbHMuamF2YQ==) | `33.33% <43.75%> (+24.99%)` | :arrow_up: |
   | [...apache/skywalking/oal/rt/parser/MetricsHolder.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-b2FwLXNlcnZlci9vYWwtcnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NreXdhbGtpbmcvb2FsL3J0L3BhcnNlci9NZXRyaWNzSG9sZGVyLmphdmE=) | `87.5% <75%> (+0.83%)` | :arrow_up: |
   | [...apm/agent/core/remote/GRPCStreamServiceStatus.java](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree#diff-YXBtLXNuaWZmZXIvYXBtLWFnZW50LWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NreXdhbGtpbmcvYXBtL2FnZW50L2NvcmUvcmVtb3RlL0dSUENTdHJlYW1TZXJ2aWNlU3RhdHVzLmphdmE=) | `41.17% <0%> (-41.18%)` | :arrow_down: |
   | ... and [10 more](https://codecov.io/gh/apache/skywalking/pull/4368/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=footer). Last update [329f7e1...3ade0f0](https://codecov.io/gh/apache/skywalking/pull/4368?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [skywalking] wu-sheng commented on issue #4368: Support Daily Index Step feature in the OAP

Posted by GitBox <gi...@apache.org>.

wu-sheng commented on issue #4368: Support Daily Index Step feature in the OAP
URL: https://github.com/apache/skywalking/pull/4368#issuecomment-586682766
 
 
   @aderm This is another challenge for the PR you are working on, also one of the reason I want to hold that PR(#4353). This will give the lower traffic user more friendly UX.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [skywalking] JaredTan95 commented on issue #4368: Support Daily Index Step feature in the OAP

Posted by GitBox <gi...@apache.org>.

JaredTan95 commented on issue #4368: Support Daily Index Step feature in the OAP
URL: https://github.com/apache/skywalking/pull/4368#issuecomment-586702568
 
 
   LGTM
   After https://github.com/apache/skywalking/pull/4214 about es index number reduced. This is another optimization.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services