[GitHub] [spark] JoshRosen commented on pull request #36680: SPARK-39283: Fixing the deadlock between TaskMemoryManager and UnsafeExternalSorter.SpillableIterator

[GitBox <gi...@apache.org>]

JoshRosen commented on PR #36680:
URL: https://github.com/apache/spark/pull/36680#issuecomment-1139135227

   I saw similar deadlocks in a user workload, where the deadlock occurred between one thread calling `UnsafeExternalSorter.cleanupResources()` and another thread which was calling `spill()` on `UnsafeExternalSorter`.
   
   I originally considered an approach which is somewhat similar to this, except I made the changes in `cleanupResources()`. I ended up searching for other approaches after I became concerned about whether my proposed change would result in other race conditions. I began exploring whether we can address this entire class of issues more fundamentally via changes to `TaskMemoryManager` itself, but I'm not 100% confident in the safety of those changes. I'll try to open a PR for discussion of that other proposal.
   
   In the meantime, let me take a closer look at this PR's fix. This looks potentially promising.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org