[jira] [Commented] (SPARK-19829) The log about driver should support rolling like executor

["hustfxj (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898713#comment-15898713 ] 

hustfxj commented on SPARK-19829:
---------------------------------

[~sowen] Yes, this is handled by systems like YARN. But my spark cluster is standalone. A standalone cluster should build its own rolling about driver' log in default other than defined log4j configuration.

> The log about driver should support rolling like executor
> ---------------------------------------------------------
>
>                 Key: SPARK-19829
>                 URL: https://issues.apache.org/jira/browse/SPARK-19829
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: hustfxj
>            Priority: Minor
>
> We should rollback the log of the driver , or the log maybe large！！！ 
> {code:title=DriverRunner.java|borderStyle=solid}
> // modify the runDriver
>   private def runDriver(builder: ProcessBuilder, baseDir: File, supervise: Boolean): Int = {
>     builder.directory(baseDir)
>     def initialize(process: Process): Unit = {
>       // Redirect stdout and stderr to files------ the old code
> //      val stdout = new File(baseDir, "stdout")
> //      CommandUtils.redirectStream(process.getInputStream, stdout)
> //
> //      val stderr = new File(baseDir, "stderr")
> //      val formattedCommand = builder.command.asScala.mkString("\"", "\" \"", "\"")
> //      val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "=" * 40)
> //      Files.append(header, stderr, StandardCharsets.UTF_8)
> //      CommandUtils.redirectStream(process.getErrorStream, stderr)
>       // Redirect its stdout and stderr to files---------support rolling
>       val stdout = new File(baseDir, "stdout")
>       stdoutAppender = FileAppender(process.getInputStream, stdout, conf)
>       val stderr = new File(baseDir, "stderr")
>       val formattedCommand = builder.command.asScala.mkString("\"", "\" \"", "\"")
>       val header = "Launch Command: %s\n%s\n\n".format(formattedCommand, "=" * 40)
>       Files.append(header, stderr, StandardCharsets.UTF_8)
>       stderrAppender = FileAppender(process.getErrorStream, stderr, conf)
>     }
>     runCommandWithRetry(ProcessBuilderLike(builder), initialize, supervise)
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org