You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2022/10/24 17:54:00 UTC
[jira] [Resolved] (SPARK-40902) Quick submission of drivers in tests to mesos scheduler results in dropping drivers
[ https://issues.apache.org/jira/browse/SPARK-40902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun resolved SPARK-40902.
-----------------------------------
Fix Version/s: 3.4.0
3.2.3
3.3.2
Assignee: Mridul Muralidharan
Resolution: Fixed
This is resolved via [https://github.com/apache/spark/pull/38378]
> Quick submission of drivers in tests to mesos scheduler results in dropping drivers
> -----------------------------------------------------------------------------------
>
> Key: SPARK-40902
> URL: https://issues.apache.org/jira/browse/SPARK-40902
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 2.4.8, 3.0.3, 3.4.0
> Reporter: Mridul Muralidharan
> Assignee: Mridul Muralidharan
> Priority: Minor
> Fix For: 3.4.0, 3.2.3, 3.3.2
>
>
> Queued drivers in MesosClusterScheduler are ordered based on MesosDriverDescription - and the default ordering checks for priority, followed by submission time. For two driver submissions with same priority and if made in quick succession (such that submission time is same due to millisecond granularity of Date), this results in dropping the second MesosDriverDescription from the queuedDrivers - as driverOrdering returns 0 when comparing the descriptions. This jira fixes the more immediate issue with tests, but we do need to relook at this for mess scheduler in general later.
> Currently, this affects tests - for example, in the latest VOTE for 3.3.1 [1] - and is not consistently reproducible unless on a fast machine.
> [1] https://lists.apache.org/thread/jof098qxp0s6qqmt9qwv52f9665b1pjg
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org