You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "wu-sheng (via GitHub)" <gi...@apache.org> on 2025/03/09 12:23:39 UTC

[I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

wu-sheng opened a new issue, #13093:
URL: https://github.com/apache/skywalking/issues/13093

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   With https://github.com/apache/skywalking/issues/12938 is going to be added in 10.2, UI and Query APIs are required to change adaptively. The cold data is long-term-persistent data in low-cost data storage, e.g. SATA, S3. Although BanyanDB API will provide nearly 100% compatible query by following the query time range to determine whether to hit on this kind of data, the reality is, the query will be much slower compared with hot and warm data.
   
   So, to be reasonable and response time more predictable, we should indicate a query to include this kind of data from UI and API perspectives. 
   
   I am proposing two kinds of API @wankai123 @hanahmily 
   1. cold-data supported. There are two conditions for this API to return true, <1> BanyanDB storage enabled, <2> BanyanDB enabled `label=cold` feature data node with proper cold data TTL.
   2. Extra flag to indicate this query for metrics/traces/logs should include cold data, we could use the term `Archived historical data` or `Long-term historical data` as the name to this flag.
   
   @Fine0830 For the UI part, we should have a check-box(or some other way) to provide this flag to the end user, and by default, NO query should enable this.
   The API <1> could determine whether UI shows this flag as other storages(JDBC and Elasticsearch) don't have such a feature.
   
   The UI style for trace and log queries is easy, we could easily add a check-box to these widgets. 
   But for metrics, the tricky point is, the query is slow. If we add this check-box to the global time-range selector, the dashboard is very hard to load. @Fine0830 You could think about how we should add this, my idea for now is, we just support historical data query in a pop-up box. For the metrics widget, we could add a menu item for it, the user could click that to pop up a query box, and in there, the user could(by the check-box checked) query data from a longer days ago.  
   
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a pull request to implement this on your own?
   
   - [ ] Yes I am willing to submit a pull request on my own!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #13093: [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage
URL: https://github.com/apache/skywalking/issues/13093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13093:
URL: https://github.com/apache/skywalking/issues/13093#issuecomment-2745952806

   Change this to milestone, we need some discussions to determine the final design.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13093:
URL: https://github.com/apache/skywalking/issues/13093#issuecomment-2745940660

   cc @Fine0830. Based on the above preference, I hope we can skip the UI change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13093:
URL: https://github.com/apache/skywalking/issues/13093#issuecomment-2745944293

   For cold data specifically, if we talk about cold traces and logs, it is hard to say, our current design is good. Because, the scale of the dataset is too huge, and the query may take minutes, we may need some kind of SQL-DB-procedure to process this long-time scanning.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13093:
URL: https://github.com/apache/skywalking/issues/13093#issuecomment-2745940045

   @wankai123 As the `warm->cold` happens after warm TTL reached, so, when we use the TTL and time in OAP to calculate the hit stages(hot, warm, cold), don't need to worry about providing the wrong stage(warm and hot are default stages, they are always picked without explicit declarance).
   I prefer to do the stage hit automatically in OAP, then we don't need query-protocol level change, and we could support native UI and PromQL/Grafana automatically.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Support cold metrics query for metrics/traces/logs in BanyanDB storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #13093:
URL: https://github.com/apache/skywalking/issues/13093#issuecomment-2719370781

   Another option is, we determine this automatically through time range. This is a kind of trade off, the good part is, this is transparent for UI and query, but the experience of the query is different. Query cold data is much slower compared with warm/hot data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org