You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by "风吹.屁屁凉" <10...@qq.com.INVALID> on 2024/01/31 08:23:04 UTC

PySpark worker CPU usage restriction

Hello everyone,

While attempting inference within a pandas_udf using Spark version 3.3, we encountered a performance bottleneck when running on a single CPU across multiple executors. The problem was effectively addressed by upgrading to Spark version 3.5. We observed that, in Spark 3.5, the Python worker's CPU core usage is restricted to 'spark.task.cpus=1'. Could you please specify which JAR file implements this feature?





Thank you.