[jira] [Assigned] (YUNIKORN-3139) Preemption throughput constrained by incorrect counter decrement logic

["Manikandan R (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Manikandan R reassigned YUNIKORN-3139:
--------------------------------------

    Assignee: Manikandan R

> Preemption throughput constrained by incorrect counter decrement logic
> ----------------------------------------------------------------------
>
>                 Key: YUNIKORN-3139
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3139
>             Project: Apache YuniKorn
>          Issue Type: Bug
>            Reporter: Victor Zhou
>            Assignee: Manikandan R
>            Priority: Critical
>
> h2. Summary
> The preemption attempt counter (`preemptAttemptsRemaining`) is decremented regardless of whether the preemption attempt succeeds or not. When a queue has many allocation asks requiring preemption, the scheduler frequently encounters asks that have already triggered preemption in previous cycles. These duplicate attempts fail precondition checks due to:
> 1. Frequency throttling: 15-second minimum interval between preemption attempts (`preemptAttemptFrequency`)
> 2. Already triggered: The ask has previously triggered preemption (`HasTriggeredPreemption()` returns true)
> Since the counter is decremented before these precondition checks, it gets wasted on asks that cannot actually trigger preemption. This severely limits preemption throughput for large-scale applications, causing multi-hour delays for applications with 1000+ pods.
> h2. How to reproduce
> Environment
> - Cluster at near-full capacity with many low-priority batch jobs
> - High-priority Spark job submitted: 1 driver + 1000+ executors
> - Sufficient preemptable resources available
> Expected Behavior: 
> - Executors should trigger preemption of low-priority jobs
> - Full allocation within 25-30 minutes
> Actual Behavior:
> - Executors took over 2 hour to allocate
> - Preemption proceeded extremely slowly despite available victims
>  
> Potential Root Cause
> Preemption throughput bottleneck:
> - maxPreemptionsPerQueue = 10 attempts per scheduling cycle
> - Counter decremented on precondition failures (no actual preemption attempted)
> https://github.com/apache/yunikorn-core/blob/master/pkg/scheduler/objects/application.go#L1017
> h2. Proposed solution
> Only decrement the counter when a preemption attempt actually occurs



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org