You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@horaedb.apache.org by "LeslieKid (via GitHub)" <gi...@apache.org> on 2024/07/17 15:51:20 UTC
[I] Tracking issue for compaction offloading [horaedb]
LeslieKid opened a new issue, #1545:
URL: https://github.com/apache/horaedb/issues/1545
### Describe This Problem
We found in production that the speed of sst compaction is unable to keep up with the speed of sst generation, leading to poor query performance. However we are unable give more resource to compaction to solve the problem because query/write is more important than compaction in the same node.
It is really hard to do a trade-off about resource allocation among query, write and compaction in lsm model. We want to compact the generated small ssts as fast as possible, but we can't tolerate its influence to query/write. And finally I think offload the compaction to the seperated nodes may be the key for it.
### Proposal
**The following is the architecture for compaction offloading.**
To support compaction offloading, we need:
- **Compaction server supporting remote compaction service (Execute Plane)**
- [ ] Introduce runnable compaction server and expose the api for horaedb node to ask for remote compaction service
- **Horaemeta supports managing the compaction servers (Control Plane)**
- [ ] Impl the ability to manage the compaction servers and collect their status (Monitor)
- [ ] Impl scheduling algorithm for load balance and expose the api for horaedb node to get the proper remote compaction node (Scheduler)
- **Horaedb node supports submitting the compaction task to remote**
- [ ] Impl remote mode compactor based on the provided compaction service and horaemeta service above
### Additional Context
This issue replaces issue #1480. Please close issue #1480 as it is outdated.
[incubator-horaedb-proto#133](https://github.com/apache/incubator-horaedb-proto/issues/133) is highly related to this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org
Re: [I] Tracking issue for compaction offloading [horaedb]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #1545:
URL: https://github.com/apache/horaedb/issues/1545#issuecomment-2330567039
I think `Horaedb node supports submitting the compaction task to remote` may be similar as `Compaction node supporting remote compaction service `.
I guess the third step can be integration test, test is very important actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org
Re: [I] Tracking issue for compaction offloading [horaedb]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #1545:
URL: https://github.com/apache/horaedb/issues/1545#issuecomment-2255969639
I think in the first stage, we can treat the `compaction cluster` as same as the `horaedb cluster`.
And we just reuse the ability about monitoring and scheduling for `horaedb cluster` in `horaemeta`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org
Re: [I] Tracking issue for compaction offloading [horaedb]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #1545:
URL: https://github.com/apache/horaedb/issues/1545#issuecomment-2336975938
Maybe you will be insterested when writing test
https://github.com/apache/horaedb/tree/main/integration_tests/dist_query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org
Re: [I] Tracking issue for compaction offloading [horaedb]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #1545:
URL: https://github.com/apache/horaedb/issues/1545#issuecomment-2233712937
looks great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org
Re: [I] Tracking issue for compaction offloading [horaedb]
Posted by "LeslieKid (via GitHub)" <gi...@apache.org>.
LeslieKid commented on issue #1545:
URL: https://github.com/apache/horaedb/issues/1545#issuecomment-2259885761
> I think in the first stage, we can treat the `compaction cluster` as same as the `horaedb cluster`. And we just reuse the ability about monitoring and scheduling for `horaedb cluster` in `horaemeta`.
>
> And we can start to desigin the specific `schedule` strategy for `compaction cluster` after it can work.
Thanks for your suggestion! I think it's a good idea to first implement a simple working version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@horaedb.apache.org
For additional commands, e-mail: commits-help@horaedb.apache.org