[jira] [Updated] (SPARK-40524) local mode with resource scheduling can hang

["Thomas Graves (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-40524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated SPARK-40524:
----------------------------------
    Summary: local mode with resource scheduling can hang  (was: local mode with resource scheduling should just fail)

> local mode with resource scheduling can hang
> --------------------------------------------
>
>                 Key: SPARK-40524
>                 URL: https://issues.apache.org/jira/browse/SPARK-40524
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.1.0
>            Reporter: Thomas Graves
>            Priority: Major
>
> If you try to run spark in local mode and request custom resources like GPU's, Spark will hang.  Resource scheduling isn't supported in local mode so just removing the request for resources fixes the issue, but its really confusing to users since it just hangs.
>  
> ie to reproduce:
> spark-sql --conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1
> Run:
> select 1
> result: hangs
> To fix run:
> spark-sql 
>  
> spark-sql> select 1;
> 1
> Time taken: 2.853 seconds, Fetched 1 row(s)
>  
> It would be nice if we just fail to start or threw an exception when using those options in local mode



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org