You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Navneet Kumar Pandey <na...@essens.no> on 2018/08/17 14:09:34 UTC
Flink not rolling log files
I am using Flink in EMR with following configuration.
{
"Classification": "flink-log4j",
"Properties": {
"log4j.logger.no":"DEBUG",
"log4j.appender.file":"org.apache.log4j.rolling.RollingFileAppender",
"log4j.appender.file.RollingPolicy.FileNamePattern":"logs/log.%d{yyyyMMdd-HHmm}.log",
"log4j.appender.file.RollingPolicy":"org.apache.log4j.rolling.TimeBasedRollingPolicy",
"log4j.appender.file.append":"false",
"log4j.appender.file.layout":"org.apache.log4j.PatternLayout",
"log4j.appender.file.layout.ConversionPattern":"%d{yyyy-MM-dd
HH:mm:ss,SSS} %-5p %-60c %x - %m%n"
}
}
FYI this configuration get written into flink's log4j.properties.As you can
see even after this setting taskmanager and jobmanager log files are not
getting rolled.
[hadoop@ip-XXXXXX ~]$ sudo ls -lh
/mnt/var/log/hadoop-yarn/containers/application_DDDDDDDDDDD_0002/container_DDDDDDDDDDDD_0002_01_000002
total 7.0G
-rw-r--r-- 1 yarn yarn 770K Aug 17 14:02 taskmanager.err
-rw-r--r-- 1 yarn yarn 6.0G Aug 17 14:02 taskmanager.log
-rw-r--r-- 1 yarn yarn 526K Aug 17 13:54 taskmanager.out
Can somebody help me to give pointer about how to roll these log files?
Note that these files are also being copied into s3.
Re: Flink not rolling log files
Posted by Dominik WosiĆski <wo...@gmail.com>.
I am using this *log4j.properties *file config for rolling files once per
day and it is working perfectly. Maybe this will give You some hint:
log4j.appender.file=org.apache.log4j.DailyRollingFileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd
HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log4j.appender.file.layout.DatePattern='.'yyyy-MM-dd
Best Regards,
Dominik.
Re: Flink not rolling log files
Posted by Gary Yao <ga...@data-artisans.com>.
Hello Navneet Kumar Pandey,
org.apache.log4j.rolling.RollingFileAppender is part of Apache Extras
Companion for Apache log4j [1]. Is that library in your classpath?
Are there hints in taskmanager.err?
Can you run:
cat /usr/lib/flink/conf/log4j.properties
on the EMR master node and show the output?
For troubleshooting, you can also try org.apache.log4j.RollingFileAppender
which can roll the file if a certain size is exceeded. An example
configuration can be found here (I have not tested it):
https://github.com/apache/flink/pull/5371/files
Best,
Gary
[1] https://logging.apache.org/log4j/extras/
On Fri, Aug 17, 2018 at 4:09 PM, Navneet Kumar Pandey <na...@essens.no>
wrote:
> I am using Flink in EMR with following configuration.
>
> {
> "Classification": "flink-log4j",
> "Properties": {
> "log4j.logger.no":"DEBUG",
> "log4j.appender.file":"org.apache.log4j.rolling.
> RollingFileAppender",
> "log4j.appender.file.RollingPolicy.FileNamePattern"
> :"logs/log.%d{yyyyMMdd-HHmm}.log",
> "log4j.appender.file.RollingPolicy":"org.apache.log4j.rolling.
> TimeBasedRollingPolicy",
> "log4j.appender.file.append":"false",
> "log4j.appender.file.layout":"org.apache.log4j.PatternLayout",
> "log4j.appender.file.layout.ConversionPattern":"%d{yyyy-MM-dd
> HH:mm:ss,SSS} %-5p %-60c %x - %m%n"
>
> }
> }
>
> FYI this configuration get written into flink's log4j.properties.As you
> can see even after this setting taskmanager and jobmanager log files are
> not getting rolled.
>
> [hadoop@ip-XXXXXX ~]$ sudo ls -lh /mnt/var/log/hadoop-yarn/
> containers/application_DDDDDDDDDDD_0002/container_
> DDDDDDDDDDDD_0002_01_000002
> total 7.0G
> -rw-r--r-- 1 yarn yarn 770K Aug 17 14:02 taskmanager.err
> -rw-r--r-- 1 yarn yarn 6.0G Aug 17 14:02 taskmanager.log
> -rw-r--r-- 1 yarn yarn 526K Aug 17 13:54 taskmanager.out
>
> Can somebody help me to give pointer about how to roll these log files?
> Note that these files are also being copied into s3.
>
>