Flink cluster log organization

[Soheil Pourbafrani <so...@gmail.com>]

Hi,

I have a Flink multinode cluster and I use Flink standalone scheduler to
deploy applications on the cluster. When I deploy applications on the
cluster I can see some log files on the path FLINK_HOME/logs will be
created but there is no separate log file for each application and all
application logs are merged into files. I have no idea how Flink organize
logs on standalone mode. I was wondering if it's possible to have a
separate log file for each application? or for example set the log file
name and path when I submit the applications on the cluster?

Re: Flink cluster log organization

[Xintong Song <to...@gmail.com>]

Hi Soheil,

Do you mean you are running multiple jobs in the same Flink standalone
cluster? Or running multiple Flink standalone clusters on the same set of
machines?

For the former, I don't think there is an easy way to separate the logs.
The log file contains logs of common framework activities that do not
belong to any particular job.

For the latter, there are two ways that I can think of to specify log file
path for a Flink cluster.

   - Set evn.log.dir in flink-conf.yaml
   - Modify log4j.appender.file.file in log4j.properties

Hope that helps.

Thank you~

Xintong Song



On Tue, May 21, 2019 at 7:03 AM Soheil Pourbafrani <so...@gmail.com>
wrote:

> Hi,
>
> I have a Flink multinode cluster and I use Flink standalone scheduler to
> deploy applications on the cluster. When I deploy applications on the
> cluster I can see some log files on the path FLINK_HOME/logs will be
> created but there is no separate log file for each application and all
> application logs are merged into files. I have no idea how Flink organize
> logs on standalone mode. I was wondering if it's possible to have a
> separate log file for each application? or for example set the log file
> name and path when I submit the applications on the cluster?
>