[jira] [Resolved] (SPARK-51660) Spark can only run one task per executor when MDC can't load

["Mridul Muralidharan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-51660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mridul Muralidharan resolved SPARK-51660.
-----------------------------------------
    Fix Version/s: 4.1.0
       Resolution: Fixed

Issue resolved by pull request 50452
[https://github.com/apache/spark/pull/50452]

> Spark can only run one task per executor when MDC can't load
> ------------------------------------------------------------
>
>                 Key: SPARK-51660
>                 URL: https://issues.apache.org/jira/browse/SPARK-51660
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 4.1.0
>            Reporter: Rob Reeves
>            Assignee: Rob Reeves
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> In SPARK-37843, logic was added to gracefully fallback to not logging with MDC when MDC cannot be initialized. The change there does not fully fix the issue. It catches NoSuchFieldError, which is thrown during MDC initialization. This will work for the first task. If a second task runs it will not throw a NoSuchFieldError. Instead it throws NoClassDefFoundError. When that happens the Spark job will hang forever.
> To fix this, the logic should be enhanced so it gracefully falls back for all tasks. We can also improve the noisiness of the logs falling back back once and disabling MDC instead of trying for every task.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org