[jira] [Created] (SPARK-40524) local mode with resource scheduling should just fail

["Thomas Graves (Jira)" <ji...@apache.org>]

Thomas Graves created SPARK-40524:
-------------------------------------

             Summary: local mode with resource scheduling should just fail
                 Key: SPARK-40524
                 URL: https://issues.apache.org/jira/browse/SPARK-40524
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.1.0
            Reporter: Thomas Graves


If you try to run spark in local mode and request custom resources like GPU's, Spark will hang.  Resource scheduling isn't supported in local mode so just removing the request for resources fixes the issue, but its really confusing to users since it just hangs.

 

ie to reproduce:

spark-sql --conf spark.executor.resource.gpu.amount=1 --conf spark.task.resource.gpu.amount=1

Run:

select 1

result: hangs

To fix run:

spark-sql 

 

spark-sql> select 1;
1
Time taken: 2.853 seconds, Fetched 1 row(s)

 

It would be nice if we just fail to start or threw an exception when using those options in local mode



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org