[GitHub] SongYadong opened a new pull request #23794: [SPARK-26884][CORE] Let task acquire memory accurately when using spilled memory

[GitBox <gi...@apache.org>]

SongYadong opened a new pull request #23794: [SPARK-26884][CORE] Let task acquire memory accurately when using spilled memory
URL: https://github.com/apache/spark/pull/23794
 
 
   ## What changes were proposed in this pull request?
   
   When task can't get required execution memory, it will call `spill()` of consumers to release more memory. After spilling, it tries again to acquire memory needed which is `(required - got)`. That's not accurate as actually `released `memory by spilling may not equal to size it try to spill. 
   So it may be better to acquire memory in more accurate size :
   - when released >= needed, acquire needed size. (`required - got`)
   - when released < needed, acquire `released `size.
   
   ## How was this patch tested?
   Existed unit tests.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org