You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "m-ghazanfar (via GitHub)" <gi...@apache.org> on 2024/02/07 04:59:32 UTC
[I] getCompactionTaskCapacity is not worker category aware. (druid)
m-ghazanfar opened a new issue, #15847:
URL: https://github.com/apache/druid/issues/15847
### Description
The [getCompactionTaskCapacity](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L321-L329) function which is used to ascertain that the Druid cluster has enough task slots before the coordinator schedules additional compaction tasks doesn't take into consideration [overlord dynamic config](https://druid.apache.org/docs/latest/configuration/#overlord-dynamic-configuration). The overlord dynamic config can prevent compaction tasks from running on specific categories of workers. This way, the compaction task capacity is incorrectly overestimated.
For example, I have two worker categories, `compaction-category` with a total of `600` task slots and another `ingestion-category` with `2000` slots(high number because of multiple ingestion task replicas).
Using the overlord dynamic config,
- `compaction-category` is configured to run the following task types,
- kill
- compact
- single_phase_sub_task
- partial_dimension_cardinality
- partial_index_generate
- partial_index_generic_merge
- `ingestion-category` is configured to run,
- index_kafka
Now, [getCompactionTaskCapacity](https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java#L321-L329) would return `2600` as the total capacity, which is inaccurate since only `600` slots are actually available for compaction tasks. While this might not pose a problem in a healthy cluster, it becomes critical during compaction task failures. The oversight leads to the coordinator creating excessive compaction tasks, resulting in contention on compaction slots and slowing down all compaction tasks. This creates a feedback loop where the increasing number of compaction tasks exacerbates contention, ultimately overwhelming the overlord with too many tasks to handle
### Affected Version
Saw this on Druid 25. Is also present in master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] getCompactionTaskCapacity is not worker category aware. (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15847:
URL: https://github.com/apache/druid/issues/15847#issuecomment-1948134115
My overlord dynamic config looks like this,
<details>
<summary> overlord dynamic config
</summary>
```json
{
"type": "equalDistributionWithCategorySpec",
"workerCategorySpec": {
"categoryMap": {
"kill": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"compact": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"single_phase_sub_task": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_dimension_cardinality": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_index_generate": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_index_generic_merge": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_range_index_generate": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"partial_dimension_distribution": {
"defaultCategory": "abc_category",
"categoryAffinity": {}
},
"index_kafka": {
"defaultCategory": "xxx_category",
"categoryAffinity": {
"datasource_2": "yyy_category",
"datasource_3": "zzz_category"
}
}
},
"strong": true
}
}
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] getCompactionTaskCapacity is not worker category aware. (druid)
Posted by "m-ghazanfar (via GitHub)" <gi...@apache.org>.
m-ghazanfar commented on issue #15847:
URL: https://github.com/apache/druid/issues/15847#issuecomment-1948141643
We worked around this by explicitly setting the `maxCompactionTaskSlots` to the total available worker slots on the compaction tier.(Actually, we set it a little higher than that to improve middle manager utilisation)
See [Update capacity for compaction tasks](https://druid.apache.org/docs/latest/api-reference/automatic-compaction-api/#update-capacity-for-compaction-tasks)
Example,
```
curl --request POST "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/compaction/taskslots?max=600"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org