[jira] [Created] (SPARK-36252) Add log files rolling policy for driver running in cluster mode with spark standalone cluster

["Jack Hu (Jira)" <ji...@apache.org>]

Jack Hu created SPARK-36252:
-------------------------------

             Summary: Add log files rolling policy for driver running in cluster mode with spark standalone cluster
                 Key: SPARK-36252
                 URL: https://issues.apache.org/jira/browse/SPARK-36252
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.1.2
            Reporter: Jack Hu


For a long running driver in cluster mode, there is no rolling policy, the log (stdout/stderr) may accupy lots of space, user needs a external tool to clean the old logs, it's not friendly. 

For executor, following 5 configurations is used to control the log file rolling policy:
{code:java}
spark.executor.logs.rolling.maxRetainedFiles
spark.executor.logs.rolling.enableCompression
spark.executor.logs.rolling.maxSize
spark.executor.logs.rolling.strategy
spark.executor.logs.rolling.time.interval
{code}

For driver running in cluster mode:
1. reuse the executor settings
2. similar to executor: add following configurations (only works for stderr/stdout for driver in cluster mode)
{code:java}
spark.driver.logs.rolling.maxRetainedFiles
spark.driver.logs.rolling.enableCompression
spark.driver.logs.rolling.maxSize
spark.driver.logs.rolling.strategy
spark.driver.logs.rolling.time.interval
{code}

#2 seems better, do you agree?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org