You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/10/08 06:10:15 UTC
[GitHub] [druid] itallam opened a new issue #11784: Parallel Minor Compaction
itallam opened a new issue #11784:
URL: https://github.com/apache/druid/issues/11784
### Description
We are currently working in adding minor compaction to our systems. We have seen with testing minor compaction, a single compaction task is working fine for small data sources. However when running minor compaction on one of our larger data sources with ~500 segments per interval, the minor compaction task is taking several hours to process. We have seen compaction jobs running for about 10 hours. This would be much too large to be of value for us. We are running a Major Compaction job after about 5 hours.
For minor compaction to be something that will work for us, we will need to reduce the runtime drastically.
To achieve that we are looking to enable parallelism for compaction. For this we are planning to implement a parallel compaction task. This task would look similar to ‘index_parallel’ as it would run multiple sub-tasks in parallel.
Each of these sub-tasks will be assigned a small sub-set of segments in the interval to be compacted. The logic within the sub tasks would be very similar to compaction/IndexTask where data is compacted and segments generated using OverrideShardSpec. Once the compaction is complete, the original segments would then be overwritten by the newly created compacted segments.
Would greatly appreciate any input and please let us know if there are any suggestions or existing solutions that we may not be aware of.
### Motivation
Please provide the following for the desired feature or change:
- A detailed description of the intended use case, if applicable
- Rationale for why the desired feature/change would be beneficial
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] itallam commented on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
itallam commented on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-981911690
@jihoonson Would you be available to discuss these issues offline? We are planning to look into the blocker issues you mentioned (https://github.com/apache/druid/issues/10155, https://github.com/apache/druid/issues/9768). I had sent an email with some details and clarifications to your apache address. Please let me know what is the best way to reach out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson edited a comment on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-942339376
@itallam Unfortunately, I'm not aware of anyone working on those issues today. Imply (the company I'm working for) is interested in promoting the minor compaction and segment locking, so maybe we could work on them in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-942339376
@itallam I'm not aware of anyone working on those issues today. Imply (the company I'm working for) is interested in promoting the minor compaction and segment locking, so maybe we could work on them in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] itallam commented on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
itallam commented on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-941506998
Thanks @jihoonson for pointing out the known issues. It looks like both of these issues are open since early 2020. Do you know if someone is working on those already and when they might be resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson edited a comment on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
jihoonson edited a comment on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-942339376
@itallam Unfortunately, I'm not aware of anyone working on those issues today. Imply (the company I'm working for) is interested in promoting the minor compaction and segment locking, so maybe we could work on them in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] itallam commented on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
itallam commented on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-941506998
Thanks @jihoonson for pointing out the known issues. It looks like both of these issues are open since early 2020. Do you know if someone is working on those already and when they might be resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-938379408
Hi @itallam, I think the minor compaction task should be able to run in parallel with multiple subtasks if you set `maxNumConcurrentSubTasks` to a larger than 1 in taskContext. However, a bigger problem for using minor compaction in production is that it's not well tested yet. There are a couple of known issues such as https://github.com/apache/druid/issues/9768, https://github.com/apache/druid/issues/10155, https://github.com/apache/druid/issues/10911. Especially for https://github.com/apache/druid/issues/10155, I believe we should fix it before using minor compaction in production. If you are using streaming ingestion, https://github.com/apache/druid/issues/9768 will be a blocker as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #11784: Parallel Minor Compaction
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #11784:
URL: https://github.com/apache/druid/issues/11784#issuecomment-942339376
@itallam I'm not aware of anyone working on those issues today. Imply (the company I'm working for) is interested in promoting the minor compaction and segment locking, so maybe we could work on them in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org