You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Fly-Style (via GitHub)" <gi...@apache.org> on 2026/06/05 13:46:03 UTC

[PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Fly-Style opened a new pull request, #19562:
URL: https://github.com/apache/druid/pull/19562

   **Description**
   
   The cost-based supervisor autoscaler wouldn't scale down a healthy, over-provisioned supervisor — one above the ideal idle ratio with low lag stayed pinned at its current task count.
   
   **Root cause.** The idle projection was linear:
   
   ```rawIdle = 1.0 - busyFraction / taskRatio;   // taskRatio = proposed / current```
   
   This assumes busy time is fully conserved when work moves onto fewer tasks, so a reasonable consolidation projects negative idle `(e.g. 1 − 0.6/0.5 =−0.2)`. That clamps to 0 (the worst point of the U-shaped idle cost) and turns an overrun into phantom virtual lag — pinning the task count even at ~0 real lag. In reality, busy grows sublinearly (an observed 2× consolidation raised busy ~1.25×, not 2×).
   
   **Fix.** Redistribute busy sublinearly:
   ```
   projectedBusy = busyFraction * (currentTaskCount / proposedTaskCount) ^ IDLE_SUBLINEARITY_EXPONENT;  // 0.32
   rawIdle = 1.0 - projectedBusy;
   ```
   `IDLE_SUBLINEARITY_EXPONENT = 0.32 (≈ log₂(1.25))` is a tuned constant based on careful testing and theorecial math application.
   
   A healthy consolidation now lands near the ideal idle ratio instead of going negative, so the supervisor scales down; the exponent stays > 0, so extreme over-consolidation still diverges and is braked. 
   
   **Validation**
   
   <details>
   Optimal task count vs. observed poll-idle ratio, across realistic configs (rate = total cluster throughput, split per-task):
   
   <img width="1200" height="750" alt="cost_based_scaledown_medium_7Mpm" src="https://github.com/user-attachments/assets/45a1387a-aa00-40b8-97a9-63796425f618" />
   
   Old version stays pinned at 128 until idle ~0.55 while new version consolidates from ~0.32.
   
   <img width="1200" height="750" alt="cost_based_scaledown_large_30Mpm" src="https://github.com/user-attachments/assets/36732e97-e622-4f1f-8257-6680378ea8d3" />
   
   Safe under load: new version consolidates earlier on the high-idle side, but at low idle both still jump to max — lag-driven scale-up is unaffected.
   
   <img width="1800" height="675" alt="cost_based_v1_vs_v2_large_30Mpm_amp0 35" src="https://github.com/user-attachments/assets/df23085a-de92-4aea-8e3d-b2791d0842ff" />
   
   The existing version is flat (pinned at max by the phantom overrun); new version consolidates and holds more tasks as lag weight rises.
   
   </details>
   
   **Release note**
   
   Fixed an issue where the cost-based supervisor autoscaler would not scale down an over-provisioned supervisor running above its ideal idle ratio with low lag.
   
     - [x] self-reviewed.
     - [x] added comments explaining the "why".
     - [x] added/updated unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3467675632


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.
+   */
+  static final double IDLE_SUBLINEARITY_EXPONENT = 0.32;

Review Comment:
   Why this specific value?
   I feel that these constants somehow make the behaviour of the auto-scaler more difficult to reason about as well as control via the weights.
   
   We should probably not be trying to predict the idleness in this manner anyway.
   
   Instead, as an alternative, we should probably consider using the `avg. processing rate * task count * idle ratio` as a measure of total work that needs to be done. Assuming that the processing rate remains the same, we can find the new idle ratio for the new task count.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3467944309


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.
+   */
+  static final double IDLE_SUBLINEARITY_EXPONENT = 0.32;

Review Comment:
   Ah, thanks!
   
   I guess you mean,
   
   > In reality, busy grows sublinearly (an observed 2× consolidation raised busy ~1.25×, not 2×).
   
   Do you mean that when task count is halved, idle ratio increases by 1.25x?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3468021797


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.
+   */
+  static final double IDLE_SUBLINEARITY_EXPONENT = 0.32;

Review Comment:
   For the alternative, I meant something along these lines
   
   ```
   current workload = total records per second = current processing rate * current task count * idle ratio
   
   assumptions:
   target workload = current workload
   predicted processing rate = current processing rate
   
   predicted idle ratio
   = target workload / (target task count * predicted processing rate)
   = current workload / (target task count * current processing rate)
   ```
   
   Let me know if that makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3468047057


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.

Review Comment:
   Please clarify here that this value was determined empirically and it was seen that when task count is halved, idle ratio increases roughly by 1.25.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "Fly-Style (via GitHub)" <gi...@apache.org>.
Fly-Style commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3467865157


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.
+   */
+  static final double IDLE_SUBLINEARITY_EXPONENT = 0.32;

Review Comment:
   > Why this specific value? 
   
   This is already explained in PR description: we want to redistribute projected busyness in a sub-linear way, basically making it U-shaped (instead of V-shaped, like it was previously).
   
   Regarding an alternative -- I'd encourage you to elaborate more, please.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3468021797


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.
+   */
+  static final double IDLE_SUBLINEARITY_EXPONENT = 0.32;

Review Comment:
   For the alternative, I meant something along these lines
   
   ```
   current workload
   = total records per second
   = current processing rate * current task count * idle ratio
   
   assumptions:
   target workload = current workload
   predicted processing rate = current processing rate
   
   predicted idle ratio
   = target workload / (target task count * predicted processing rate)
   = current workload / (target task count * current processing rate)
   ```
   
   Let me know if that makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "Fly-Style (via GitHub)" <gi...@apache.org>.
Fly-Style commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3473593969


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "Fly-Style (via GitHub)" <gi...@apache.org>.
Fly-Style merged PR #19562:
URL: https://github.com/apache/druid/pull/19562


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

Re: [PR] feat: enhance U-shape idle prediction for scale-down scenarios (druid)

Posted by "Fly-Style (via GitHub)" <gi...@apache.org>.
Fly-Style commented on code in PR #19562:
URL: https://github.com/apache/druid/pull/19562#discussion_r3467865157


##########
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/autoscaler/WeightedCostFunction.java:
##########
@@ -42,6 +42,12 @@ public class WeightedCostFunction
    */
   static final double LAG_AMPLIFICATION_MULTIPLIER = 0.4;
 
+  /**
+   * Exponent (&lt; 1) for sublinear busy redistribution in the idle projection: busy grows as
+   * {@code (currentTaskCount / proposedTaskCount)^EXPONENT}, not linearly. Calibrated as log2(1.25) ~= 0.32.
+   */
+  static final double IDLE_SUBLINEARITY_EXPONENT = 0.32;

Review Comment:
   > Why this specific value? 
   
   This is already explained in PR description: we want to redistribute projected busyness in a sub-linear way, basically making it U-shaped (instead of V-shaped, like it was previously). It is also proved by theoretical math behind it (see plots).
   
   Regarding an alternative -- I'd encourage you to elaborate more, please.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org