You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/09/26 10:26:53 UTC

[GitHub] [druid] tejaswini-imply opened a new pull request, #13144: Adds Idle feature to `SeekableStreamSupervisor` for inactive stream

tejaswini-imply opened a new pull request, #13144:
URL: https://github.com/apache/druid/pull/13144

### Description
When a stream becomes inactive i.e. no new data arrives and all existing data is caught up, there is no need for Supervisor to create new indexing tasks. This feature uses `LagBasedAutoScaler` to turn Supervisor idle when stream is inactive for configured amount of time (See conf section).

- LagBasedAutoScaler submits either `IdleNotice` or `DynamicAllocationTasksNotice` based on activity in the stream every `LagBasedAutoScalerConfig.scaleActionPeriodMillis`.
- When Supervisor is in idle state it just doesn't create any new indexing tasks.
- This temporary IDLE state is cleared when supervisor is restarted e.g. updated spec.
- While the supervisor is in the temporary idle state, the lag-based autoscaler will continue to run, checking for any changes in the latest offsets (or if lag is > 0). If activity in the stream is detected again, then the autoscaler will submit another notice to move the supervisor out of its idle state and resume creating indexing tasks.

Please note the initial delay before new data is ingested from when new data enters into the stream is frequency of idle/dynamic task allocation notice submission (default - 1 min) + initial warmup time to create new indexing tasks.

#### How current offsets fetching work:
- Supervisor is not idle: current offsets are fetched from running tasks.
- Supervisor is idle: current offsets are fetched from metadata storage since no tasks are likely running.

### Configuration changes
- `druid.supervisor.enableIdleBehaviour` overlord property enables this feature. Default value is false.
- `LagBasedAutoScalerConfig.minPauseSupervisorIfStreamIdleMillis` - Minimum time interval to wait until stream is considered inactive/idle. Default value is 60,000 millis.
- Adds new State - `IDLE` - to Basic states of Supervisor.

<hr>

##### Key changed/added classes in this PR
* `SeekableStreamSupervisor`
* `KafkaSupervisor`
* `KinesisSupervisor`
* `SupervisorStateManager`
* `LagBasedAutoScaler`

<hr>

This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org