You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Attila Zsolt Piros (Jira)" <ji...@apache.org> on 2021/11/18 18:38:00 UTC

[jira] [Updated] (SPARK-37373) Collect LocalSparkContext worker logs in case of test failure

     [ https://issues.apache.org/jira/browse/SPARK-37373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Attila Zsolt Piros updated SPARK-37373:
---------------------------------------
    Affects Version/s: 3.3.0
                           (was: 3.2.0)

> Collect LocalSparkContext worker logs in case of test failure
> -------------------------------------------------------------
>
>                 Key: SPARK-37373
>                 URL: https://issues.apache.org/jira/browse/SPARK-37373
>             Project: Spark
>          Issue Type: Improvement
>          Components: Tests
>    Affects Versions: 3.3.0
>            Reporter: Attila Zsolt Piros
>            Assignee: Attila Zsolt Piros
>            Priority: Major
>
> About 50 test suites are using LocalSparkContext by specifying "local-cluster" as the cluster URL. In this case executor logs will be under the worker dir which is a temp directory and as such will be deleted at shutdown (for details see https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70)
> So when a test fails and the error was on the executor side the log will be lost. 
> This is only for local cluster tests and not for standalone tests where logs will be kept in the "<SPARK_HOME>/work".  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org