You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Tian Gao (Jira)" <ji...@apache.org> on 2025/12/08 20:03:00 UTC

[jira] [Created] (SPARK-54640) Replace select.select with select.poll in worker

Tian Gao created SPARK-54640:
--------------------------------

             Summary: Replace select.select with select.poll in worker
                 Key: SPARK-54640
                 URL: https://issues.apache.org/jira/browse/SPARK-54640
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Tian Gao


We use select.select in worker.py to communicate with daemon. However, select.select has a limit that it can't deal with fd > 1024. This might be an issue on a heavy load system. We should use the more modern select.poll to do this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org