[I] Tracking issue for compaction offload [incubator-horaedb]

["Rachelint (via GitHub)" <gi...@apache.org>]

Rachelint opened a new issue, #1480:
URL: https://github.com/apache/incubator-horaedb/issues/1480

   ### Describe This Problem
   
   We found in production that the speed of sst compaction is unable to keep up with the speed of sst generation, leading to poor query performance... However we are unable give more resource to compaction to solve the problem because query/write is more important than compaction in the same node.
   It is really hard to do a trade-off about resource allocation among query, write and compaction in lsm model... We want to compact the generated small ssts as fast as possible, but we can't tolerate its influence to query/write. And finally I think offload the compaction to the seperated nodes may be the key for it.
   
   
   ### Proposal
   
   For supporting compaction offload, we need:
   
   - Special node supporting remote compaction service
   
     - [ ] Impl compaction service
   
   - Horaedb node supports submitting the real compaction node to remote
   
     - [ ] Refactor the compaction process and define necessary traits
     - [ ] Impl remote mode compactor based on traits above
   
   - Horaemeta supports managing the special compaction nodes
     
     - [ ] Impl the ability to manage the compaction nodes
     - [ ] Expose the api for horaedb node to get the proper remote compaction ndoe
   
   ### Additional Context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org

Re: [I] Tracking issue for compaction offload [horaedb]

["Rachelint (via GitHub)" <gi...@apache.org>]

Rachelint closed issue #1480: Tracking issue for compaction offload
URL: https://github.com/apache/horaedb/issues/1480


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org