[GitHub] [spark] cloud-fan opened a new pull request #24269: [SPARK-27341][CORE] fix a deadlock between TaskMemoryManager and UnsafeExternalSorter

[GitBox <gi...@apache.org>]

cloud-fan opened a new pull request #24269: [SPARK-27341][CORE] fix a deadlock between TaskMemoryManager and UnsafeExternalSorter
URL: https://github.com/apache/spark/pull/24269
 
 
   ## What changes were proposed in this pull request?
   
   This is a long-standing bug.
   
   In `TaskMemoryManager.acquireExecutionMemory`, we may lock `TaskMemoryManager` and spill `MemoryConsumer`s. `UnsafeExternalSorter` is a `MemoryConsumer` and locks its `SpillableIterator` during spill.
   
   In `UnsafeExternalSorter#SpillableIterator.loadNext`, we lock `SpillableIterator` and may call `freePage` which locks `TaskMemoryManager`.
   
   If there are 2 threads doing these 2 locking chains together, a deadlock happens:
   thread1: lock `TaskMemoryManager` and then `SpillableIterator`
   thread2: lock `SpillableIterator` and then `TaskMemoryManager`
   
   As an example, PythonUDFExec launches 2 threads for one task, which can trigger this deadlock.
   
   The thread dump when the deadlock happens:
   thread 1
   ```
   org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$SpillableIterator.spill(UnsafeExternalSorter.java:535)
   org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.spill(UnsafeExternalSorter.java:201)
   org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:177) => holding Monitor(org.apache.spark.memory.TaskMemoryManager@49054460})
   org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:285)
   org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:117)
   org.apache.spark.sql.execution.python.HybridRowQueue.createNewQueue(RowQueue.scala:227)
   org.apache.spark.sql.execution.python.HybridRowQueue.add(RowQueue.scala:250)
   org.apache.spark.sql.execution.python.EvalPythonExec$
   ```
   thread 2
   ```
   org.apache.spark.memory.TaskMemoryManager.freePage(TaskMemoryManager.java:334)
   org.apache.spark.memory.MemoryConsumer.freePage(MemoryConsumer.java:130)
   org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.access$1100(UnsafeExternalSorter.java:49)
   org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$SpillableIterator.loadNext(UnsafeExternalSorter.java:593) => holding Monitor(org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter$SpillableIterator@717107972})
   org.apache.spark.sql.execution.UnsafeExternalRowSorter$1.next(UnsafeExternalRowSorter.java:187)
   org.apache.spark.sql.execution.UnsafeExternalRowSorter$1.next(UnsafeExternalRowSorter.java:174)
   org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage5.processNext(Unknown Source)
   
   ```
   
   ## How was this patch tested?
   
   manual test, by inserting some sync points in the code, to enter to code flow that triggers dead lock.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org