[jira] [Commented] (FLINK-26425) Yarn aggregate log files in a rolling fashion for flink

["Prabhu Joseph (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717324#comment-17717324 ] 

Prabhu Joseph commented on FLINK-26425:
---------------------------------------

This is very useful to Flink on YARN, as without this, the rolled log files (jobmanager.log.n or taskmanager.log.n) won't be aggregated into the HDFS/S3 Log Aggregation Path, which leads to a local disk full. [SPARK-15990|https://issues.apache.org/jira/browse/SPARK-15990] has added this support.

{quote}Could you give some examples of the rolling fashion under the settings of spark.yarn.rolledLog.excludePattern and spark.yarn.rolledLog.includePattern?
{quote}
Example include and exclude pattern will look like below. YARN will aggregate all rotated jobmanager log files and won't touch the out file.

flink.yarn.rolledLog.includePattern=jobmanager.log.*
flink.yarn.rolledLog.excludePattern=jobmanager.out

I can come up with a patch for this. Could you assign this to me? Thanks.

> Yarn aggregate log files in a rolling fashion for flink
> -------------------------------------------------------
>
>                 Key: FLINK-26425
>                 URL: https://issues.apache.org/jira/browse/FLINK-26425
>             Project: Flink
>          Issue Type: Improvement
>          Components: Deployment / YARN
>    Affects Versions: 1.14.3
>            Reporter: wang da wei
>            Priority: Major
>
> For the spark on yarn, you can set the spark.yarn.rolledLog.includePattern、spark.yarn.rolledLog.excludePattern to filter the log files and those log files will be aggregated in a rolling fashion,Do you have similar Settings in Flink？



--
This message was sent by Atlassian Jira
(v8.20.10#820010)