You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2024/12/31 02:11:22 UTC

[I] [Feature] Improve Concurrency of Merging Operations in Data and Index Storage [skywalking]

hanahmily opened a new issue, #12914:
URL: https://github.com/apache/skywalking/issues/12914

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   The data and index storage components currently have independent merge scheduling strategies. This approach can lead to many concurrent merging operations—sometimes exceeding 10 simultaneous merges.
   
   When heavy merging tasks occur among these operations, CPU and memory usage can spike significantly, potentially impacting system stability and performance.
   
   To address this issue, I propose introducing a **concurrency barrier** as a global controller for all merging schedulers. This barrier will limit the number of concurrent merging operations across both data and index storage, effectively reducing the likelihood of resource contention and performance degradation.
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a pull request to implement this on your own?
   
   - [ ] Yes I am willing to submit a pull request on my own!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Improve Concurrency of Merging Operations in Data and Index Storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #12914:
URL: https://github.com/apache/skywalking/issues/12914#issuecomment-2585475642

   Hi, you could start with testing. This is not an easy work, unless you have been familiar witb new DB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Improve Concurrency of Merging Operations in Data and Index Storage [skywalking]

Posted by "Placidoe (via GitHub)" <gi...@apache.org>.
Placidoe commented on issue #12914:
URL: https://github.com/apache/skywalking/issues/12914#issuecomment-2585344774

   hi, I'm interested in this, can I try it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Improve Concurrency of Merging Operations in Data and Index Storage [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #12914: [Feature] Improve Concurrency of Merging Operations in Data and Index Storage
URL: https://github.com/apache/skywalking/issues/12914


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org