You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hongze Zhang (Jira)" <ji...@apache.org> on 2024/08/07 00:54:00 UTC
[jira] [Updated] (SPARK-49133) Make MemoryConsumer thread-safe
[ https://issues.apache.org/jira/browse/SPARK-49133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hongze Zhang updated SPARK-49133:
---------------------------------
Description:
TaskMemoryManager is thread-safe however MemoryConsumer is not.
Thinking of multiple threads are allocating memory in the same task (although it's known not to be ideal regarding Spark's one-thread-per-task model), it leads to practices that developer should add a lock in the caller code of MemoryConsumer. However since it's common that one memory consumer spills another, using lock in user code could easily causes ABBA dead lock, e.g., consumer A locks it self to acquire memory from task memory manager, while consumer B locks task memory manager then spills A.
It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so far is to make accesses of field `used` atomic.
was:
TaskMemoryManager is thread-safe however MemoryConsumer is not.
Thinking of multiple threads are allocating memory in the same task (although it's known not to be ideal regarding Spark's one-thread-per-task model), it leads to issues that developer should add a lock in the caller code of MemoryConsumer. However since it's common that one memory consumer spills another, using lock in user code could easily causes ABBA dead lock, e.g., consumer A locks it self to acquire memory from task memory manager, while consumer B locks task memory manager then spills A.
It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so far is to make accesses of field `used` atomic.
> Make MemoryConsumer thread-safe
> -------------------------------
>
> Key: SPARK-49133
> URL: https://issues.apache.org/jira/browse/SPARK-49133
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: Hongze Zhang
> Priority: Major
>
> TaskMemoryManager is thread-safe however MemoryConsumer is not.
> Thinking of multiple threads are allocating memory in the same task (although it's known not to be ideal regarding Spark's one-thread-per-task model), it leads to practices that developer should add a lock in the caller code of MemoryConsumer. However since it's common that one memory consumer spills another, using lock in user code could easily causes ABBA dead lock, e.g., consumer A locks it self to acquire memory from task memory manager, while consumer B locks task memory manager then spills A.
> It's easy to make MemoryConsumer thread-safe. AFAICS the only change needed so far is to make accesses of field `used` atomic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org