[jira] [Resolved] (SPARK-37373) Collect LocalSparkContext worker logs in case of test failure

["Gengliang Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-37373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gengliang Wang resolved SPARK-37373.
------------------------------------
    Fix Version/s: 3.3.0
       Resolution: Fixed

Issue resolved by pull request 34651
[https://github.com/apache/spark/pull/34651]

> Collect LocalSparkContext worker logs in case of test failure
> -------------------------------------------------------------
>
>                 Key: SPARK-37373
>                 URL: https://issues.apache.org/jira/browse/SPARK-37373
>             Project: Spark
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 3.3.0
>            Reporter: Attila Zsolt Piros
>            Assignee: Attila Zsolt Piros
>            Priority: Major
>             Fix For: 3.3.0
>
>
> About 50 test suites are using LocalSparkContext by specifying "local-cluster" as the cluster URL. In this case executor logs will be under the worker dir which is a temp directory and as such will be deleted at shutdown (for details see https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70)
> So when a test fails and the error was on the executor side the log will be lost. 
> This is only for local cluster tests and not for standalone tests where logs will be kept in the "<SPARK_HOME>/work".  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org