[GitHub] [spark] srowen commented on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python

[GitBox <gi...@apache.org>]

srowen commented on issue #25545: [SPARK-28843][PYTHON] Set OMP_NUM_THREADS to executor cores for python
URL: https://github.com/apache/spark/pull/25545#issuecomment-524214318
 
 
   Yes I think this is a good change. I didn't realize it caused memory pressure with numpy (possibly other usages?) I've even generally seen OMP_NUM_THREADS=1 because won't each Python task run in a separate Python process? If you have 4 cores, you don't want 4 processes each running 4 threads. That said I don't have much understanding of how OMP_NUM_THREADS works. Bringing a default down from "all the machine's cores" is better.
   
   Do you want to only set this env var if it's not already set, to let people override it?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org