You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/04/16 09:00:21 UTC
[GitHub] [druid] yuanlihan opened a new issue #9712: Enable auto minor
compaction
yuanlihan opened a new issue #9712: Enable auto minor compaction
URL: https://github.com/apache/druid/issues/9712
### Description
Note that @jihoonson introduced the minor compaction feature to Druid at version 0.16.x, which helps to effectively compact small segments. As we know, the cluster may suffer from the too many small segments issue if there are big pipelines that have a long trickle of late data.
But after upgrading to version 0.16.1, I found that the `auto minor compaction` was not implemented then. But still I rushed to the feature by making an adaptation based on the implementation of minor compaction and it works well till now. Thanks @jihoonson and all others involved with this greate feature.
Nowadays, we decide to upgrade to newer version, but find that the `auto minor compaction` is still not available. May I know if @jihoonson or anyone else WIP about this feature now? And if not, maybe I can have a try to raise a PR about the the adaptation.
PS: there was a detail discuss https://github.com/apache/druid/issues/8489 between @jihoonson and @himanshug about `CompactionTask` and the `auto compaction`.
### Motivation
- It will be more convenient to issue minor compaction tasks if enabling `auto minor compaction`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9712: Enable auto minor
compaction
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9712: Enable auto minor compaction
URL: https://github.com/apache/druid/issues/9712#issuecomment-614975399
Hi @yuanlihan, thank you for trying out minor compaction!
The thing is that you can enable auto minor compaction now by setting `forceTimeChunkLock = false` in the tuningConfig of coordinator compaction configuration. However, with auto compaction, you can use only either minor compaction or major compaction for now. To support both major and minor compactions, `CompactSegments` should be improved to be able to run both of them.
Also, please note that the segment lock which is used in minor compaction is still experimental and there are a couple of known issues such as https://github.com/apache/druid/issues/9571.
Probably I could work on fixing those bugs with segment lock and improving auto compaction to support both major and minor compactions at some point this year, but am unsure when it could be yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org