You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "wuyi (Jira)" <ji...@apache.org> on 2021/05/06 15:22:00 UTC

[jira] [Created] (SPARK-35330) Tune shuffle requests frequency dynamically in the case of Netty OOM

wuyi created SPARK-35330:
----------------------------

             Summary: Tune shuffle requests frequency dynamically in the case of Netty OOM
                 Key: SPARK-35330
                 URL: https://issues.apache.org/jira/browse/SPARK-35330
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.2.0
            Reporter: wuyi


In [https://github.com/apache/spark/pull/32287,] the PR proposes to use a flag to indicate the Netty OOM status and defer the fetch requests when OOM happens. However, it doesn't change the request frequency after OOM happens. Thus, it's possible to hit the OOM again later if the deferred requests are still too much to fetch concurrently. Therefore, tuning the fetch frequency might be a good idea to solve the issue. Please see the detailed discussion at https://github.com/apache/spark/pull/32287#discussion_r625287419 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org