You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2024/02/28 01:05:16 UTC

[I] [Feature] Implement Hot-Warm-Cold Data Management Architecture [skywalking]

hanahmily opened a new issue, #11945:
URL: https://github.com/apache/skywalking/issues/11945

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   As BanyanDB grows and deals with bigger datasets, it's important to manage data efficiently and optimize storage to keep costs down. Right now, BanyanDB doesn't differentiate between "hot" data that gets accessed frequently and "cold" data that's barely touched.
   
   After discussing with @HHoflittlefish777,  we propose the implementation of a hot-warm-cold data management architecture within BanyanDB. This architecture would categorize data based on access patterns and move them between different storage tiers:
   - **Hot**: Data that is frequently accessed and requires fast read/write speeds. This should be stored on the fastest storage media available.
   - **Warm**: Data that is accessed less frequently but may still need to be readily available. This could be stored on lower-cost, lower-speed media. _It applies a force merge to optimize the data files._
   - **Cold**: Rarely accessed data that can be archived, preventing any changes from being made to it. This should be stored on the cheapest storage media, such as cold HDDs or even offsite/cloud storage.
   
   Contributor: @HHoflittlefish777
   
   ## Next Steps
   - [ ] Discuss the viability and scope of the proposed feature
   - [ ] Draft a detailed design document
   - [ ] Review and approve the design document
   - [ ] Implement the feature
   - [ ] Perform thorough testing, including performance and resilience tests
   - [ ] Update documentation to reflect the new architecture
   - [ ] Create tutorials or guides for users to understand how to work with the new data tiers
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a pull request to implement this on your own?
   
   - [ ] Yes I am willing to submit a pull request on my own!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Implement Hot-Warm-Cold Data Management Architecture [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #11945:
URL: https://github.com/apache/skywalking/issues/11945#issuecomment-2547515524

   This feat relies on #12876 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Implement Hot-Warm-Cold Data Management Architecture [skywalking]

Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on issue #11945:
URL: https://github.com/apache/skywalking/issues/11945#issuecomment-1968030211

   please assign to me :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Implement Hot-Warm-Cold Data Management Architecture [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #11945:
URL: https://github.com/apache/skywalking/issues/11945#issuecomment-2456032600

   @hanahmily I reassigned this back to you, as the previous request has no update for over half of a year.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Implement Hot-Warm-Cold Data Management Architecture [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily closed issue #11945: [Feature] Implement Hot-Warm-Cold Data Management Architecture
URL: https://github.com/apache/skywalking/issues/11945


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org