You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "vogievetsky (via GitHub)" <gi...@apache.org> on 2023/03/28 20:58:32 UTC
[GitHub] [druid] vogievetsky opened a new pull request, #13993: Web console: add a nice UI for overlord dynamic configs and improve the docs
vogievetsky opened a new pull request, #13993:
URL: https://github.com/apache/druid/pull/13993
Add a nice form (with JSON fallback) to the overlord dynamic config dialog
<img width="621" alt="image" src="https://user-images.githubusercontent.com/177816/228364506-3cdd1597-b71b-4781-beff-4099dcde0cee.png">
To do this I had to read the docs and found them lacking. I asked @kfaraz to help me make heads and tails of the logic. Here was his summary that I rewrote in the docs.
Here are his original notes (screenshot form Slack DM):
<img width="845" alt="image" src="https://user-images.githubusercontent.com/177816/228364978-0b2fddfc-31b2-4c0d-b83d-1ac72c73d5af.png">
If there are better way to word it I am all ears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on a diff in pull request #13993: Web console: add a nice UI for overlord dynamic configs and improve the docs
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #13993:
URL: https://github.com/apache/druid/pull/13993#discussion_r1152737075
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
Review Comment:
```suggestion
Issuing a GET request to the same URL returns the current Overlord dynamic config.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
Review Comment:
```suggestion
There are 4 options for select strategies:
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
+- a preferred worker (a worker listed in `affinityConfig`) if available
+- not assigned at all (remains pending) if preferred workers are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
-###### Equal Distribution
+- any worker if no categoryConfig is given for task type
+- any worker if categoryConfig is given for task type but no category is given for datasource and there's no default category
+- a preferred worker (based on categoryConfig and category for datasource) if available
+- any worker if categoryConfig and category are given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all if preferred workers are not available and `categoryConfig` is `strong`
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+In both cases, Druid constructs the eligible worker and selects one depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
Review Comment:
nit:
```suggestion
In both the cases, Druid determines the list of eligible workers and selects one depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
Review Comment:
```suggestion
To view the last `n` entries of the audit history of worker config, issue a GET request to the following URL:
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
+- a preferred worker (a worker listed in `affinityConfig`) if available
Review Comment:
this item might be better placed before the previous item in the list
```suggestion
- a preferred worker listed in the `affinityConfig` for this datasource if it has available capacity
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
+- a preferred worker (a worker listed in `affinityConfig`) if available
+- not assigned at all (remains pending) if preferred workers are not available and affinity is `strong`
Review Comment:
```suggestion
- no worker if preferred workers are not available and affinity is _strong_ i.e. `strong: true`. In this case, the task remains in "pending" state. The chosen provisioning strategy (e.g. `pendingTaskBased`) may then use the total number of pending tasks to determine if a new node should be provisioned.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
+- a preferred worker (a worker listed in `affinityConfig`) if available
+- not assigned at all (remains pending) if preferred workers are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
Review Comment:
Similar suggestions here as with `affinityConfig`:
```suggestion
If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies), then a task of a given datasource may be assigned to:
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
+- a preferred worker (a worker listed in `affinityConfig`) if available
+- not assigned at all (remains pending) if preferred workers are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
-###### Equal Distribution
+- any worker if no categoryConfig is given for task type
+- any worker if categoryConfig is given for task type but no category is given for datasource and there's no default category
+- a preferred worker (based on categoryConfig and category for datasource) if available
+- any worker if categoryConfig and category are given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all if preferred workers are not available and `categoryConfig` is `strong`
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+In both cases, Druid constructs the eligible worker and selects one depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
+
+If you are using auto-scaling, use the `fillCapacity` select strategy since auto-scaled nodes can
+not be assigned a category, and you want the work to be concentrated on the fewest number of workers to allow the empty ones to scale down.
+
+###### `equalDistribution`
+
+Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running.
+This evenly distributes work across your MiddleManagers.
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`equalDistribution`.|required; must be `equalDistribution`|
-|`affinityConfig`|[Affinity config](#affinity) object|null (no affinity)|
+|`type`|`equalDistribution`|required; must be `equalDistribution`|
+|`affinityConfig`|[Affinity config](#affinityconfig) object|null (no affinity)|
-###### Equal Distribution With Category Spec
+###### `equalDistributionWithCategorySpec`
-This strategy is a variant of `Equal Distribution`, which support `workerCategorySpec` field rather than `affinityConfig`. By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the tasks' **taskType** and **dataSource name**. This strategy can't work with `AutoScaler` since the behavior is undefined.
+This strategy is a variant of `equalDistribution`, which supports `workerCategorySpec` field rather than `affinityConfig`.
+By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the task's **type** and **dataSource**.
Review Comment:
nit:
```suggestion
By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the **type** and **dataSource** of the task.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
Review Comment:
```suggestion
At a high level, the select strategy determines the list of eligible workers for a given task using
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
+- a preferred worker (a worker listed in `affinityConfig`) if available
+- not assigned at all (remains pending) if preferred workers are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
-###### Equal Distribution
+- any worker if no categoryConfig is given for task type
+- any worker if categoryConfig is given for task type but no category is given for datasource and there's no default category
+- a preferred worker (based on categoryConfig and category for datasource) if available
+- any worker if categoryConfig and category are given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all if preferred workers are not available and `categoryConfig` is `strong`
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+In both cases, Druid constructs the eligible worker and selects one depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
+
+If you are using auto-scaling, use the `fillCapacity` select strategy since auto-scaled nodes can
Review Comment:
We should probably emphasize this point a little by putting it in a note or warn box or something.
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
+- a non-affinity worker if preferred workers are not available and affinity is `weak`
Review Comment:
I think we should either italicize "weak" (since it is really code or config), or we can use `strong: false`.
```suggestion
- a non-affinity worker if preferred workers are not available and the affinity is _weak_ i.e. `strong: false`.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,86 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
+either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+There are 4 options for select strategies:
+
+- [`equalDistribution`](#equaldistribution)
+- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
+- [`fillCapacity`](#fillcapacity)
+- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
+
+A `javascript` option is also available but should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
Review Comment:
Slight rewording of the later part of this sentence and the list items for a little more readability:
```suggestion
If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies), then a task of a given datasource may be assigned to:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 317brian commented on a diff in pull request #13993: Web console: add a nice UI for overlord dynamic configs and improve the docs
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #13993:
URL: https://github.com/apache/druid/pull/13993#discussion_r1152522377
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
-###### Equal Distribution
+In either case, after the eligible worker list is constructed one will be selected depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+If you are using auto-scaling it only makes sense to use the `fillCapacity` select strategy since auto-scaled nodes can
+not be assigned a category, and you want the work to be concentrated on the fewest number of workers to allow the empty ones to scale down.
+
+###### `equalDistribution`
+
+Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running.
+This will evenly distribute work across your MiddleManagers.
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`equalDistribution`.|required; must be `equalDistribution`|
+|`type`|`equalDistribution`|required; must be `equalDistribution`|
|`affinityConfig`|[Affinity config](#affinity) object|null (no affinity)|
Review Comment:
```suggestion
|`affinityConfig`|[Affinity config](#affinityconfig) object|null (no affinity)|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] fjy merged pull request #13993: Web console: add a nice UI for overlord dynamic configs and improve the docs
Posted by "fjy (via GitHub)" <gi...@apache.org>.
fjy merged PR #13993:
URL: https://github.com/apache/druid/pull/13993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on pull request #13993: Web console: add a nice UI for overlord dynamic configs and improve the docs
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #13993:
URL: https://github.com/apache/druid/pull/13993#issuecomment-1490168571
Tried out the new console changes, they are very easy to use now!
Encountered a minor bug. The content in the suggestion box seems to overflow in some cases.
<img width="605" alt="Screenshot 2023-03-30 at 5 17 34 PM" src="https://user-images.githubusercontent.com/18635897/228826984-772fd75a-1b26-445a-853c-3aa871e279bf.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] 317brian commented on a diff in pull request #13993: Web console: add a nice UI for overlord dynamic configs and improve the docs
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #13993:
URL: https://github.com/apache/druid/pull/13993#discussion_r1151181175
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
Review Comment:
```suggestion
Issuing a GET request at the same URL returns the current Overlord dynamic config spec.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
Review Comment:
```suggestion
|`selectStrategy`| Describes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
Review Comment:
```suggestion
At a high level, the select strategy determines the list of possible workers that a task can be assigned to using
either an `affinityConfig` or a `categorySpec`. Then, Druid assigns the task by either trying to distribute load equally
(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
Review Comment:
```suggestion
A `javascript` option is also available but should only be used for prototyping new strategies.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
Review Comment:
```suggestion
- a non-affinity worker if no affinity is specified for that datasource. Any worker not listed in the `affinityConfig` is considered a non-affinity worker.
- a non-affinity worker if preferred workers are not available and affinity is `weak`
- a preferred worker (a worker listed in `affinityConfig`) if available
- not assigned at all (remains pending) if preferred MiddleManagers are not available and affinity is `strong`
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
Review Comment:
```suggestion
If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
Review Comment:
```suggestion
There are 4 options for select strategies:
- [`equalDistribution`](#equaldistribution)
- [`equalDistributionWithCategorySpec`](#equalDistributionWithCategorySpec)
- [`fillCapacity`](#fillcapacity)
- [`fillCapacityWithCategorySpec`](#fillcapacitywithcategoryspec)
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
Review Comment:
```suggestion
If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task, the list of workers eligible to be assigned is determined as follows:
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
Review Comment:
```suggestion
- any worker if no categoryConfig is given for task type
- any worker if categoryConfig is given for task type but no category is given for datasource and there's no default category
- a preferred worker (based on categoryConfig and category for datasource) if available
- any worker if categoryConfig and category are given but no preferred worker is available and categoryConfig is `weak`
- not assigned at all if preferred workers are not available and `categoryConfig` is `strong`
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
-###### Equal Distribution
+In either case, after the eligible worker list is constructed one will be selected depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+If you are using auto-scaling it only makes sense to use the `fillCapacity` select strategy since auto-scaled nodes can
Review Comment:
```suggestion
If you are using auto-scaling, use the `fillCapacity` select strategy since auto-scaled nodes can
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
-###### Equal Distribution
+In either case, after the eligible worker list is constructed one will be selected depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
Review Comment:
```suggestion
In both cases, Druid constructs the eligible worker and selects one depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
-###### Equal Distribution
+In either case, after the eligible worker list is constructed one will be selected depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+If you are using auto-scaling it only makes sense to use the `fillCapacity` select strategy since auto-scaled nodes can
+not be assigned a category, and you want the work to be concentrated on the fewest number of workers to allow the empty ones to scale down.
+
+###### `equalDistribution`
+
+Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running.
+This will evenly distribute work across your MiddleManagers.
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`equalDistribution`.|required; must be `equalDistribution`|
+|`type`|`equalDistribution`|required; must be `equalDistribution`|
|`affinityConfig`|[Affinity config](#affinity) object|null (no affinity)|
-###### Equal Distribution With Category Spec
+###### `equalDistributionWithCategorySpec`
-This strategy is a variant of `Equal Distribution`, which support `workerCategorySpec` field rather than `affinityConfig`. By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the tasks' **taskType** and **dataSource name**. This strategy can't work with `AutoScaler` since the behavior is undefined.
+This strategy is a variant of `equalDistribution`, which support `workerCategorySpec` field rather than `affinityConfig`.
+By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the tasks' **type** and **dataSource**.
+This strategy can't work with `AutoScaler` since the behavior is undefined.
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`equalDistributionWithCategorySpec`.|required; must be `equalDistributionWithCategorySpec`|
+|`type`|`equalDistributionWithCategorySpec`|required; must be `equalDistributionWithCategorySpec`|
|`workerCategorySpec`|[Worker Category Spec](#workercategoryspec) object|null (no worker category spec)|
-Example: specify tasks default to run on **c1** whose task
-type is "index_kafka", while dataSource "ds1" run on **c2**.
+Example: for tasks of type "index_kafka" default to running on middle managers of category `c1`, except tasks writing to datasource "ds1" which should run on middle managers of category `c2`.
Review Comment:
```suggestion
Example: tasks of type "index_kafka" default to running on MiddleManagers of category `c1`, except for tasks that write to datasource "ds1," which run on MiddleManagers of category `c2`.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
-###### Equal Distribution
+In either case, after the eligible worker list is constructed one will be selected depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+If you are using auto-scaling it only makes sense to use the `fillCapacity` select strategy since auto-scaled nodes can
+not be assigned a category, and you want the work to be concentrated on the fewest number of workers to allow the empty ones to scale down.
+
+###### `equalDistribution`
+
+Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running.
+This will evenly distribute work across your MiddleManagers.
Review Comment:
```suggestion
This evenly distributes work across your MiddleManagers.
```
##########
docs/configuration/index.md:
##########
@@ -1223,52 +1223,80 @@ A sample worker config spec is shown below:
}
```
-Issuing a GET request at the same URL will return the current worker config spec that is currently in place. The worker config spec list above is just a sample for EC2 and it is possible to extend the code base for other deployment environments. A description of the worker config spec is shown below.
+Issuing a GET request at the same URL will return the current Overlord dynamic config spec.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`selectStrategy`|How to assign tasks to MiddleManagers. Choices are `fillCapacity`, `equalDistribution`, and `javascript`.|equalDistribution|
-|`autoScaler`|Only used if autoscaling is enabled. See below.|null|
+|Property| Description | Default |
+|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|
+|`selectStrategy`| Desctibes how to assign tasks to MiddleManagers. The type can be `equalDistribution`, `equalDistributionWithCategorySpec`, `fillCapacity`, `fillCapacityWithCategorySpec`, and `javascript`. | `{"type":"equalDistribution"} |
+|`autoScaler`| Only used if autoscaling is enabled. See below. | null |
To view the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?interval=<interval>
```
-default value of interval can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
+The default value of `interval` can be specified by setting `druid.audit.manager.auditHistoryMillis` (1 week if not configured) in Overlord runtime.properties.
To view last `n` entries of the audit history of worker config issue a GET request to the URL -
```
http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker/history?count=<n>
```
-##### Worker Select Strategy
+##### Worker select strategy
+
+The select strategy controls how Druid assigns tasks to workers (MiddleManagers).
+At a high level a select strategy determines the list of possible workers that the task can be assigned to using
+either an `affinityConfig` or a `categorySpec` and then it assigns the task by either trying to distribute load equally
+(`equalDistribution`) or to fill as many workers as possible to capacity (`fillCapacity`).
+This forms 4 possible options for supported select strategies.
+A `javascript` option is also available which should only be used for prototyping new strategies.
+
+If an `affinityConfig` is provided (as part of `fillCapacity` and `equalDistribution` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
+
+- a non-affinity worker, if no affinity is specified for that datasource (any worker not listed in the `affinityConfig` is known as a "Non-affinity worker")
+- a non-affinity worker, if preferred workers are not available and affinity is `weak`
+- a preferred worker (based on affinityConfig), if available
+- not assigned at all (i.e. remains pending), if preferred MMs are not available and affinity is `strong`
+
+Note that every worker listed in the `affinityConfig` will only be used for the assigned datasources and no other.
+
+If a `categorySpec` is provided (as part of `fillCapacityWithCategorySpec` and `equalDistributionWithCategorySpec` strategies) for a given task the list of workers eligible to be assigned is determined as follows:
-Worker select strategies control how Druid assigns tasks to MiddleManagers.
+- any worker, if no categoryConfig given for task type
+- any worker, if categoryConfig given for task type but no category given for datasource and no default category either
+- a preferred worker (based on categoryConfig + category for datasource), if available
+- any worker, if categoryConfig given and category given but no preferred worker is available and categoryConfig is `weak`
+- not assigned at all, if preferred workers are not available and `categoryConfig` is `strong`
-###### Equal Distribution
+In either case, after the eligible worker list is constructed one will be selected depending on their load with the goal of either distributing the load equally or filling as few workers as possible.
-Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running. This is useful if
-you want work evenly distributed across your MiddleManagers.
+If you are using auto-scaling it only makes sense to use the `fillCapacity` select strategy since auto-scaled nodes can
+not be assigned a category, and you want the work to be concentrated on the fewest number of workers to allow the empty ones to scale down.
+
+###### `equalDistribution`
+
+Tasks are assigned to the MiddleManager with the most free slots at the time the task begins running.
+This will evenly distribute work across your MiddleManagers.
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`equalDistribution`.|required; must be `equalDistribution`|
+|`type`|`equalDistribution`|required; must be `equalDistribution`|
|`affinityConfig`|[Affinity config](#affinity) object|null (no affinity)|
-###### Equal Distribution With Category Spec
+###### `equalDistributionWithCategorySpec`
-This strategy is a variant of `Equal Distribution`, which support `workerCategorySpec` field rather than `affinityConfig`. By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the tasks' **taskType** and **dataSource name**. This strategy can't work with `AutoScaler` since the behavior is undefined.
+This strategy is a variant of `equalDistribution`, which support `workerCategorySpec` field rather than `affinityConfig`.
+By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the tasks' **type** and **dataSource**.
+This strategy can't work with `AutoScaler` since the behavior is undefined.
Review Comment:
```suggestion
This strategy is a variant of `equalDistribution`, which supports `workerCategorySpec` field rather than `affinityConfig`.
By specifying `workerCategorySpec`, you can assign tasks to run on different categories of MiddleManagers based on the task's **type** and **dataSource**.
This strategy doesn't work with `AutoScaler` since the behavior is undefined.
```
##########
docs/configuration/index.md:
##########
@@ -1278,45 +1306,45 @@ type is "index_kafka", while dataSource "ds1" run on **c2**.
"strong": false,
"categoryMap": {
"index_kafka": {
- "defaultCategory": "c1",
- "categoryAffinity": {
- "ds1": "c2"
- }
+ "defaultCategory": "c1",
+ "categoryAffinity": {
+ "ds1": "c2"
+ }
}
}
}
}
}
```
-###### Fill Capacity
+###### `fillCapacity`
-Tasks are assigned to the worker with the most currently-running tasks at the time the task begins running. This is
-useful in situations where you are elastically auto-scaling MiddleManagers, since it will tend to pack some full and
+Tasks are assigned to the worker with the most currently-running tasks. This is
+useful when you are auto-scaling MiddleManagers, since it will tend to pack some full and
leave others empty. The empty ones can be safely terminated.
Note that if `druid.indexer.runner.pendingTasksRunnerNumThreads` is set to _N_ > 1, then this strategy will fill _N_
MiddleManagers up to capacity simultaneously, rather than a single MiddleManager.
-|Property|Description|Default|
-|--------|-----------|-------|
-|`type`|`fillCapacity`.|required; must be `fillCapacity`|
-|`affinityConfig`|[Affinity config](#affinity) object|null (no affinity)|
+|Property| Description |Default|
+|--------|-----------------------------------------|-------|
+|`type`| `fillCapacity` |required; must be `fillCapacity`|
+|`affinityConfig`| [AffinityConfig](#affinityconfig) object |null (no affinity)|
-###### Fill Capacity With Category Spec
+###### `fillCapacityWithCategorySpec`
-This strategy is a variant of `Fill Capacity`, which support `workerCategorySpec` field rather than `affinityConfig`. The usage is the same with _equalDistributionWithCategorySpec_ strategy. This strategy can't work with `AutoScaler` since the behavior is undefined.
+This strategy is a variant of `fillCapacity`, which supports `workerCategorySpec` instead of an `affinityConfig`.
+The usage is the same with _equalDistributionWithCategorySpec_ strategy.
+This strategy can't work with `AutoScaler` since the behavior is undefined.
Review Comment:
```suggestion
The usage is the same as `equalDistributionWithCategorySpec` strategy.
This strategy doesn't work with `AutoScaler` since the behavior is undefined.
```
##########
docs/configuration/index.md:
##########
@@ -1278,45 +1306,45 @@ type is "index_kafka", while dataSource "ds1" run on **c2**.
"strong": false,
"categoryMap": {
"index_kafka": {
- "defaultCategory": "c1",
- "categoryAffinity": {
- "ds1": "c2"
- }
+ "defaultCategory": "c1",
+ "categoryAffinity": {
+ "ds1": "c2"
+ }
}
}
}
}
}
```
-###### Fill Capacity
+###### `fillCapacity`
-Tasks are assigned to the worker with the most currently-running tasks at the time the task begins running. This is
-useful in situations where you are elastically auto-scaling MiddleManagers, since it will tend to pack some full and
+Tasks are assigned to the worker with the most currently-running tasks. This is
+useful when you are auto-scaling MiddleManagers, since it will tend to pack some full and
Review Comment:
```suggestion
Tasks are assigned to the worker with the most currently-running tasks. This is
useful when you are auto-scaling MiddleManagers since it tends to pack some full and
```
##########
docs/configuration/index.md:
##########
@@ -1808,7 +1838,7 @@ This laning strategy is best suited for cases where one or more external applica
##### Server Configuration
-Druid uses Jetty to serve HTTP requests. Each query being processed consumes a single thread from `druid.server.http.numThreads`, so consider defining `druid.query.scheduler.numThreads` to a lower value in order to reserve HTTP threads for responding to health checks, lookup loading, and other non-query, and in most cases comparatively very short lived, HTTP requests.
+Druid uses Jetty to serve HTTP requests. Each query being processed consumes a single thread from `druid.server.http.numThreads`, so consider defining `druid.query.scheduler.numThreads` to a lower value in order to reserve HTTP threads for responding to health checks, lookup loading, and other non-query, and in most cases comparatively very short-lived, HTTP requests.
Review Comment:
```suggestion
Druid uses Jetty to serve HTTP requests. Each query being processed consumes a single thread from `druid.server.http.numThreads`, so consider defining `druid.query.scheduler.numThreads` to a lower value in order to reserve HTTP threads for responding to health checks, lookup loading, and other non-query, (in most cases) comparatively very short-lived, HTTP requests.
```
##########
docs/configuration/index.md:
##########
@@ -1326,32 +1354,33 @@ its better to write a druid extension module with extending current worker selec
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`javascript`.|required; must be `javascript`|
+|`type`|`javascript`|required; must be `javascript`|
|`function`|String representing JavaScript function| |
Example: a function that sends batch_index_task to workers 10.0.0.1 and 10.0.0.2 and all other tasks to other available workers.
```
{
-"type":"javascript",
-"function":"function (config, zkWorkers, task) {\nvar batch_workers = new java.util.ArrayList();\nbatch_workers.add(\"middleManager1_hostname:8091\");\nbatch_workers.add(\"middleManager2_hostname:8091\");\nworkers = zkWorkers.keySet().toArray();\nvar sortedWorkers = new Array()\n;for(var i = 0; i < workers.length; i++){\n sortedWorkers[i] = workers[i];\n}\nArray.prototype.sort.call(sortedWorkers,function(a, b){return zkWorkers.get(b).getCurrCapacityUsed() - zkWorkers.get(a).getCurrCapacityUsed();});\nvar minWorkerVer = config.getMinWorkerVersion();\nfor (var i = 0; i < sortedWorkers.length; i++) {\n var worker = sortedWorkers[i];\n var zkWorker = zkWorkers.get(worker);\n if(zkWorker.canRunTask(task) && zkWorker.isValidVersion(minWorkerVer)){\n if(task.getType() == 'index_hadoop' && batch_workers.contains(worker)){\n return worker;\n } else {\n if(task.getType() != 'index_hadoop' && !batch_workers.contains(worker)){\n return worker;\n }\n }\n }\n}\nr
eturn null;\n}"
+ "type":"javascript",
+ "function":"function (config, zkWorkers, task) {\nvar batch_workers = new java.util.ArrayList();\nbatch_workers.add(\"middleManager1_hostname:8091\");\nbatch_workers.add(\"middleManager2_hostname:8091\");\nworkers = zkWorkers.keySet().toArray();\nvar sortedWorkers = new Array()\n;for(var i = 0; i < workers.length; i++){\n sortedWorkers[i] = workers[i];\n}\nArray.prototype.sort.call(sortedWorkers,function(a, b){return zkWorkers.get(b).getCurrCapacityUsed() - zkWorkers.get(a).getCurrCapacityUsed();});\nvar minWorkerVer = config.getMinWorkerVersion();\nfor (var i = 0; i < sortedWorkers.length; i++) {\n var worker = sortedWorkers[i];\n var zkWorker = zkWorkers.get(worker);\n if(zkWorker.canRunTask(task) && zkWorker.isValidVersion(minWorkerVer)){\n if(task.getType() == 'index_hadoop' && batch_workers.contains(worker)){\n return worker;\n } else {\n if(task.getType() != 'index_hadoop' && !batch_workers.contains(worker)){\n return worker;\n }\n }\n }\n}\
nreturn null;\n}"
}
```
> JavaScript-based functionality is disabled by default. Please refer to the Druid [JavaScript programming guide](../development/javascript.md) for guidelines about using Druid's JavaScript functionality, including instructions on how to enable it.
-###### Affinity
+###### AffinityConfig
-Use the `affinityConfig` field to pass affinity configuration to the _equalDistribution_ and _fillCapacity_ strategies. If not provided, the default is to not use affinity at all.
+Use the `affinityConfig` field to pass affinity configuration to the `equalDistribution` and `fillCapacity` strategies.
+If not provided, the default is to have no affinity.
|Property|Description|Default|
|--------|-----------|-------|
-|`affinity`|JSON object mapping a datasource String name to a list of indexing service MiddleManager host:port String values. Druid doesn't perform DNS resolution, so the 'host' value must match what is configured on the MiddleManager and what the MiddleManager announces itself as (examine the Overlord logs to see what your MiddleManager announces itself as).|{}|
+|`affinity`|JSON object mapping a datasource String name to a list of indexing service MiddleManager `host:port` values. Druid doesn't perform DNS resolution, so the 'host' value must match what is configured on the MiddleManager and what the MiddleManager announces itself as (examine the Overlord logs to see what your MiddleManager announces itself as).|{}|
|`strong`|When `true` tasks for a datasource must be assigned to affinity-mapped MiddleManagers. Tasks remain queued until a slot becomes available. When `false`, Druid may assign tasks for a datasource to other MiddleManagers when affinity-mapped MiddleManagers are unavailable to run queued tasks.|false|
###### WorkerCategorySpec
Review Comment:
```suggestion
###### workerCategorySpec
```
##########
docs/configuration/index.md:
##########
@@ -1326,32 +1354,33 @@ its better to write a druid extension module with extending current worker selec
|Property|Description|Default|
|--------|-----------|-------|
-|`type`|`javascript`.|required; must be `javascript`|
+|`type`|`javascript`|required; must be `javascript`|
|`function`|String representing JavaScript function| |
Example: a function that sends batch_index_task to workers 10.0.0.1 and 10.0.0.2 and all other tasks to other available workers.
```
{
-"type":"javascript",
-"function":"function (config, zkWorkers, task) {\nvar batch_workers = new java.util.ArrayList();\nbatch_workers.add(\"middleManager1_hostname:8091\");\nbatch_workers.add(\"middleManager2_hostname:8091\");\nworkers = zkWorkers.keySet().toArray();\nvar sortedWorkers = new Array()\n;for(var i = 0; i < workers.length; i++){\n sortedWorkers[i] = workers[i];\n}\nArray.prototype.sort.call(sortedWorkers,function(a, b){return zkWorkers.get(b).getCurrCapacityUsed() - zkWorkers.get(a).getCurrCapacityUsed();});\nvar minWorkerVer = config.getMinWorkerVersion();\nfor (var i = 0; i < sortedWorkers.length; i++) {\n var worker = sortedWorkers[i];\n var zkWorker = zkWorkers.get(worker);\n if(zkWorker.canRunTask(task) && zkWorker.isValidVersion(minWorkerVer)){\n if(task.getType() == 'index_hadoop' && batch_workers.contains(worker)){\n return worker;\n } else {\n if(task.getType() != 'index_hadoop' && !batch_workers.contains(worker)){\n return worker;\n }\n }\n }\n}\nr
eturn null;\n}"
+ "type":"javascript",
+ "function":"function (config, zkWorkers, task) {\nvar batch_workers = new java.util.ArrayList();\nbatch_workers.add(\"middleManager1_hostname:8091\");\nbatch_workers.add(\"middleManager2_hostname:8091\");\nworkers = zkWorkers.keySet().toArray();\nvar sortedWorkers = new Array()\n;for(var i = 0; i < workers.length; i++){\n sortedWorkers[i] = workers[i];\n}\nArray.prototype.sort.call(sortedWorkers,function(a, b){return zkWorkers.get(b).getCurrCapacityUsed() - zkWorkers.get(a).getCurrCapacityUsed();});\nvar minWorkerVer = config.getMinWorkerVersion();\nfor (var i = 0; i < sortedWorkers.length; i++) {\n var worker = sortedWorkers[i];\n var zkWorker = zkWorkers.get(worker);\n if(zkWorker.canRunTask(task) && zkWorker.isValidVersion(minWorkerVer)){\n if(task.getType() == 'index_hadoop' && batch_workers.contains(worker)){\n return worker;\n } else {\n if(task.getType() != 'index_hadoop' && !batch_workers.contains(worker)){\n return worker;\n }\n }\n }\n}\
nreturn null;\n}"
}
```
> JavaScript-based functionality is disabled by default. Please refer to the Druid [JavaScript programming guide](../development/javascript.md) for guidelines about using Druid's JavaScript functionality, including instructions on how to enable it.
-###### Affinity
+###### AffinityConfig
Review Comment:
```suggestion
###### affinityConfig
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org