You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Daniel Fischer <d....@gmail.com> on 2022/07/13 10:44:15 UTC

Kubernetes Operator - Logging

Hi,

I have a job running in Kubernetes that I'm looking to migrate to the new
Kubernetes Operator
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/.
My question is about logging. My job currently uses logback, but reading
through the documentation I'm not seeing a way to use anything other than
log4j. My questions are:

1. Is it possible to use a logging framework other than log4j with the
kubernetes operator?
2. Is it possible to configure logging through the FlinkDeployment? The
documentation seems to read that logging must be configured when installing
the operator.
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#logging

Thanks!

Re: Kubernetes Operator - Logging

Posted by Daniel Fischer <d....@gmail.com>.
Biao and Matyas,

Thanks for the responses. Biao, I tried what you suggested, but maybe I'm
not clear on this step

Use -Dlogback.configurationFile=<file> to specify the logback.xml(mounted
or packaged in the image) in spec.flinkConfiguration

I added my logback-console.xml to /opt/flink/logback-console.xml in my
image, and specified

spec:
    flinkConfiguration:
        logback.configurationFile: /opt/flink/logback-console.xml

However I still get the following error on startup

ERROR in ch.qos.logback.classic.joran.JoranConfigurator@3abfe836 - Could
not open URL [file:/opt/flink/conf/logback-console.xml].
java.io.FileNotFoundException: /opt/flink/conf/logback-console.xml (No such
file or directory)

I tried adding logback-console.xml to /opt/flink/conf, but it seems that
the operator wipes that directory except for flink-conf.yaml and
log4j-console.properties, which is why I just added it to /opt/flink
instead.

Thanks! Dan

On Wed, Jul 13, 2022 at 7:36 AM Őrhidi Mátyás <ma...@gmail.com>
wrote:

> Hi Daniel,
>
> Some additional info on the current approach we have in the operator:
>
> Manipulating plain property files was relatively easy from the Helm chart,
> hence we went down this route first. We could try to document at least an
> example for logback then see if we can improve the setup further.
>
> Best,
> Matyas
>
> On Wed, Jul 13, 2022 at 1:21 PM Geng Biao <bi...@gmail.com> wrote:
>
>> Hi Daniel,
>>
>>
>>
>> I believe it is possible to use logback for jobs submitted by flink
>> kubernetes operator but I have not found an out-of-box solution either.
>> I tried following steps:
>>
>>    - Create custom flink image following
>>    https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
>>    to add necessary jars/xmls and use the custom image
>>    - Use -Dlogback.configurationFile=<file> to specify the
>>    logback.xml(mounted or packaged in the image) in spec.flinkConfiguration
>>    - Submit the job
>>
>>
>>
>> Best,
>>
>> Biao Geng
>>
>>
>>
>> *From: *Daniel Fischer <d....@gmail.com>
>> *Date: *Wednesday, July 13, 2022 at 6:44 PM
>> *To: *user@flink.apache.org <us...@flink.apache.org>
>> *Subject: *Kubernetes Operator - Logging
>>
>> Hi,
>>
>>
>>
>> I have a job running in Kubernetes that I'm looking to migrate to the new
>> Kubernetes Operator
>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/.
>> My question is about logging. My job currently uses logback, but reading
>> through the documentation I'm not seeing a way to use anything other than
>> log4j. My questions are:
>>
>>
>>
>> 1. Is it possible to use a logging framework other than log4j with the
>> kubernetes operator?
>>
>> 2. Is it possible to configure logging through the FlinkDeployment? The
>> documentation seems to read that logging must be configured when installing
>> the operator.
>> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#logging
>>
>>
>>
>> Thanks!
>>
>

Re: Kubernetes Operator - Logging

Posted by Őrhidi Mátyás <ma...@gmail.com>.
Hi Daniel,

Some additional info on the current approach we have in the operator:

Manipulating plain property files was relatively easy from the Helm chart,
hence we went down this route first. We could try to document at least an
example for logback then see if we can improve the setup further.

Best,
Matyas

On Wed, Jul 13, 2022 at 1:21 PM Geng Biao <bi...@gmail.com> wrote:

> Hi Daniel,
>
>
>
> I believe it is possible to use logback for jobs submitted by flink
> kubernetes operator but I have not found an out-of-box solution either.
> I tried following steps:
>
>    - Create custom flink image following
>    https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
>    to add necessary jars/xmls and use the custom image
>    - Use -Dlogback.configurationFile=<file> to specify the
>    logback.xml(mounted or packaged in the image) in spec.flinkConfiguration
>    - Submit the job
>
>
>
> Best,
>
> Biao Geng
>
>
>
> *From: *Daniel Fischer <d....@gmail.com>
> *Date: *Wednesday, July 13, 2022 at 6:44 PM
> *To: *user@flink.apache.org <us...@flink.apache.org>
> *Subject: *Kubernetes Operator - Logging
>
> Hi,
>
>
>
> I have a job running in Kubernetes that I'm looking to migrate to the new
> Kubernetes Operator
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/.
> My question is about logging. My job currently uses logback, but reading
> through the documentation I'm not seeing a way to use anything other than
> log4j. My questions are:
>
>
>
> 1. Is it possible to use a logging framework other than log4j with the
> kubernetes operator?
>
> 2. Is it possible to configure logging through the FlinkDeployment? The
> documentation seems to read that logging must be configured when installing
> the operator.
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#logging
>
>
>
> Thanks!
>

Re: Kubernetes Operator - Logging

Posted by Geng Biao <bi...@gmail.com>.
Hi Daniel,

I believe it is possible to use logback for jobs submitted by flink kubernetes operator but I have not found an out-of-box solution either.
I tried following steps:

  *   Create custom flink image following https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback to add necessary jars/xmls and use the custom image
  *   Use -Dlogback.configurationFile=<file> to specify the logback.xml(mounted or packaged in the image) in spec.flinkConfiguration
  *   Submit the job

Best,
Biao Geng

From: Daniel Fischer <d....@gmail.com>
Date: Wednesday, July 13, 2022 at 6:44 PM
To: user@flink.apache.org <us...@flink.apache.org>
Subject: Kubernetes Operator - Logging
Hi,

I have a job running in Kubernetes that I'm looking to migrate to the new Kubernetes Operator https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/. My question is about logging. My job currently uses logback, but reading through the documentation I'm not seeing a way to use anything other than log4j. My questions are:

1. Is it possible to use a logging framework other than log4j with the kubernetes operator?
2. Is it possible to configure logging through the FlinkDeployment? The documentation seems to read that logging must be configured when installing the operator. https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/metrics-logging/#logging

Thanks!