You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by "Wilfred Spiegelenburg (Jira)" <ji...@apache.org> on 2023/11/12 23:57:00 UTC
[jira] [Resolved] (YUNIKORN-2141) Should not preempt placeholders which has been released
[ https://issues.apache.org/jira/browse/YUNIKORN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wilfred Spiegelenburg resolved YUNIKORN-2141.
---------------------------------------------
Fix Version/s: 1.4.0
Resolution: Fixed
> Should not preempt placeholders which has been released
> --------------------------------------------------------
>
> Key: YUNIKORN-2141
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2141
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: core - scheduler
> Reporter: Qi Zhu
> Assignee: Qi Zhu
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.4.0
>
>
> The details about the bug:
> * The real pod created and waiting for scheduling after placeholders bound
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:14.912Z\tINFO\tcache/task_state.go:380\tTask state transition\t{\"app\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"task\": \"8837de6e-d888-4549-9baf-254c8a807421\", \"taskAlias\": \"dex-app-q5nslqd5/ogautaealleventsdynamicu2klogfm2-50-eb8bde8baf814091-driver\", \"source\": \"New\", \"destination\": \"Pending\", \"event\": \"InitTask\"}"}{code}
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:14.912Z\tINFO\tcache/task_state.go:380\tTask state transition\t{\"app\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"task\": \"8837de6e-d888-4549-9baf-254c8a807421\", \"taskAlias\": \"dex-app-q5nslqd5/ogautaealleventsdynamicu2klogfm2-50-eb8bde8baf814091-driver\", \"source\": \"Pending\", \"destination\": \"Scheduling\", \"event\": \"SubmitTask\"}"}{code}
> * Scheduler replace placeholder processed, and send release allocation request to shim side:
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:14.912Z\tINFO\tscheduler/partition.go:828\tscheduler replace placeholder processed\t{\"appID\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"allocationKey\": \"8837de6e-d888-4549-9baf-254c8a807421\", \"uuid\": \"9508439d-60a2-404e-9c84-bd2c6783b5c7\", \"placeholder released uuid\": \"cc243ba1-7054-4b07-8344-6afb1424b1e0\"}"}{code}
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:14.913Z\tINFO\tcache/application.go:637\ttry to release pod from application\t{\"appID\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"allocationUUID\": \"cc243ba1-7054-4b07-8344-6afb1424b1e0\", \"terminationType\": \"PLACEHOLDER_REPLACED\"}"}{code}
> * The same time, Preempting task try to preempt the already sent release allocation
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:20.870Z\tINFO\tobjects/preemption.go:563\tPreempting task\t{\"applicationID\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"allocationKey\": \"e6e91651-7152-42f5-8504-355590fa0079\", \"nodeID\": \"ip-10-157-240-201.ec2.internal\", \"resources\": \"map[memory:3430940672 pods:1 vcore:2100]\"}"}{code}
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:20.871Z\tINFO\tcache/application.go:637\ttry to release pod from application\t{\"appID\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"allocationUUID\": \"cc243ba1-7054-4b07-8344-6afb1424b1e0\", \"terminationType\": \"PREEMPTED_BY_SCHEDULER\"}"}{code}
> * The pod deleted and trigger complete task and the terminationType is PREEMPTED_BY_SCHEDULER
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:45.489Z\tINFO\tgeneral/general.go:204\tdelete pod\t{\"appType\": \"general\", \"namespace\": \"dex-app-q5nslqd5\", \"podName\": \"tg-spark-driver-spark-28105bdfe17b494887c0c4-0\", \"podUID\": \"e6e91651-7152-42f5-8504-355590fa0079\"}"}{code}
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:45.489Z\tINFO\tcache/task_state.go:380\tTask state transition\t{\"app\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"task\": \"e6e91651-7152-42f5-8504-355590fa0079\", \"taskAlias\": \"dex-app-q5nslqd5/tg-spark-driver-spark-28105bdfe17b494887c0c4-0\", \"source\": \"Bound\", \"destination\": \"Completed\", \"event\": \"CompleteTask\"}"}{code}
> {code:java}
> {"stream":"stdout","log":"2023-11-08T15:16:45.489Z\tINFO\tscheduler/partition.go:1245\tremoving allocation from application\t{\"appID\": \"spark-28105bdfe17b494887c0c443f8a3ab0f\", \"allocationId\": \"cc243ba1-7054-4b07-8344-6afb1424b1e0\", \"terminationType\": \"PREEMPTED_BY_SCHEDULER\"}"}{code}
> * The real pod will always pending, because the core side doesn't receive the release response for PLACEHOLDER_REPLACED
> {code:java}
> // if we have an uuid the termination type is important
> if release.TerminationType == si.TerminationType_PLACEHOLDER_REPLACED {
> log.Logger().Info("replacing placeholder allocation",
> zap.String("appID", appID),
> zap.String("allocationId", uuid))
> if alloc := app.ReplaceAllocation(uuid); alloc != nil {
> released = append(released, alloc)
> }
> } else {
> log.Logger().Info("removing allocation from application",
> zap.String("appID", appID),
> zap.String("allocationId", uuid),
> zap.String("terminationType", release.TerminationType.String()))
> if alloc := app.RemoveAllocation(uuid); alloc != nil {
> released = append(released, alloc)
> }
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org