You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/01/31 14:25:30 UTC
[GitHub] tgravescs commented on issue #23672: [SPARK-26750]Estimate memory
overhead with multi-cores
tgravescs commented on issue #23672: [SPARK-26750]Estimate memory overhead with multi-cores
URL: https://github.com/apache/spark/pull/23672#issuecomment-459362695
I agree that with the # of cores going up we might need more overhead, but I think this is to aggressive. #cores * 0.10, if I have 10 cores that is equivalent to the total amount requested for the heap. ie I ask for 30GiB heap, I won't need 30GiB of overhead. I think this is especially true on the driver/AM side. As @srowen said, generally as you increase the cores you increase the memory which increases your overhead as well.
I think ideally we have some performance or real world numbers showing what the % increase will be.
You have this SHARED_MEMORY_OVERHEAD_MIN, did that number come from somewhere specific?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org