Logging for both JM and TM in the web UI

[Nikola Hrusov <n....@gmail.com>]

Hi,

I am trying to deploy my flink application and get some logs on the web UI.
I am using the latest version of flink (1.14). I am deploying it with
docker.

When I run the application without any custom log4j2 everything works:

   - I can see the logs in "Logs" tab on the web UI for both JM and TM
   - I can see the logs in "Log List" tab named
   "flink--standalonesession-0-jm.log" and "flink--taskexecutor-0-tm.log" for
   JM and TM respectively. I can also see those files inside the containers
   in /opt/flink/log directory



Now I want to have my custom naming. I have named the log files "flink.log"
for both TM and JM
I have copy/pasted the log4j-console.properties from flink and I have only
changed lines 55-56 from:

*appender.rolling.fileName = ${sys:log.file}*

*appender.rolling.filePattern = ${sys:log.file}.%i*


to

*appender.rolling.fileName = /opt/flink/log/flink.log*

*appender.rolling.filePattern = /opt/flink/log/flink.log.%*i



When I do that:

   - I do not see logs in "Logs" tab on the web UI for neither JM nor TM
   - I can see logs in "Log List" tab named "flink.log" only for my TM. I
   cannot see the the logs for the JM.



If I set "web.log.path: /opt/flink/log/flink.log" for both JM and TM, then
I can see logs for TM under "Log List" tab.
But they do not show in the "Logs" tab for TM and they do not show for the
JM at all. I can find the logs in /opt/flink/log for both TM and JM.


My question is: How can I show the logs in the UI under both "Logs" and
"Log List" tabs when I use a custom name for my log file?


Here is a very simple example to be reproduced:
https://github.com/nikobearrr/flink-logging-webui
To switch between default logging and "custom" logging you need to comment
out the volume part of both services and remove the "web.log.path"
environmental variable.

Regards,
Nikola

Re: Logging for both JM and TM in the web UI

[Caizhi Weng <ts...@gmail.com>]

Hi!

There is no need to change log4j configs or *web.log.path*. Just add
*log.file* to your JVM options of both JM and TM (by setting env.java.opts:
-Dlog.file=/opt/flink/log/flink.log to your Flink config) and it should
work.

For more information on why it works see [1]. You can see
*appender.rolling.fileName
= ${sys:log.file}*.

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions

Nikola Hrusov <n....@gmail.com> 于2022年2月7日周一 01:56写道：

> Hi,
>
> I am trying to deploy my flink application and get some logs on the web
> UI. I am using the latest version of flink (1.14). I am deploying it with
> docker.
>
> When I run the application without any custom log4j2 everything works:
>
>    - I can see the logs in "Logs" tab on the web UI for both JM and TM
>    - I can see the logs in "Log List" tab named
>    "flink--standalonesession-0-jm.log" and "flink--taskexecutor-0-tm.log" for
>    JM and TM respectively. I can also see those files inside the
>    containers in /opt/flink/log directory
>
>
>
> Now I want to have my custom naming. I have named the log files
> "flink.log" for both TM and JM
> I have copy/pasted the log4j-console.properties from flink and I have only
> changed lines 55-56 from:
>
> *appender.rolling.fileName = ${sys:log.file}*
>
> *appender.rolling.filePattern = ${sys:log.file}.%i*
>
>
> to
>
> *appender.rolling.fileName = /opt/flink/log/flink.log*
>
> *appender.rolling.filePattern = /opt/flink/log/flink.log.%*i
>
>
>
> When I do that:
>
>    - I do not see logs in "Logs" tab on the web UI for neither JM nor TM
>    - I can see logs in "Log List" tab named "flink.log" only for my TM. I
>    cannot see the the logs for the JM.
>
>
>
> If I set "web.log.path: /opt/flink/log/flink.log" for both JM and TM, then
> I can see logs for TM under "Log List" tab.
> But they do not show in the "Logs" tab for TM and they do not show for the
> JM at all. I can find the logs in /opt/flink/log for both TM and JM.
>
>
> My question is: How can I show the logs in the UI under both "Logs" and
> "Log List" tabs when I use a custom name for my log file?
>
>
> Here is a very simple example to be reproduced:
> https://github.com/nikobearrr/flink-logging-webui
> To switch between default logging and "custom" logging you need to comment
> out the volume part of both services and remove the "web.log.path"
> environmental variable.
>
> Regards,
> Nikola
>