You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by houssem <me...@gmail.com> on 2021/08/31 11:27:50 UTC
logback variable substitution in kubernetes
Hello,
I am running a flink application cluster in standalone kubernetes mode and i a using logback
as a logging framework , th problem is i am not able tu use environment variables configured in my pod inside my logback-console.xml file .
I copied this file from my file system while building my image.
Dockerfile:
..
COPY logback-console.xml $FLINK_HOME/conf/
..
here is my logback-console file::
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
<logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
<logger name="akka" level="$FLINK_LOG_LEVEL" />
<logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
<logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
####Appenders####
#file
<appender name="file" class="ch.qos.logback.core.FileAppender">
<file>$LOG_FILE</file>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="net.logstash.logback.layout.LogstashLayout">
<suffix class="ch.qos.logback.classic.PatternLayout">
<pattern>%n</pattern>
</suffix>
</layout>
</encoder>
</appender>
#console
<appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
<!-- encoders are assigned the type
ch.qos.logback.classic.encoder.PatternLayoutEncoder by default -->
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="net.logstash.logback.layout.LogstashLayout">
<suffix class="ch.qos.logback.classic.PatternLayout">
<pattern>%n</pattern>
</suffix>
</layout>
</encoder>
</appender>
<root level="$LOG_LEVEL">
<appender-ref ref="file"/>
<appender-ref ref="stdout"/>
</root>
Re: logback variable substitution in kubernetes
Posted by houssem <me...@gmail.com>.
Hello,
i tried with the curly braces, unfortunately it didn't work same thing .
On 2021/09/01 17:54:58, Alexis Sarda-Espinosa <al...@microfocus.com> wrote:
> I'm fairly certain you need the curly braces surrounding the variable, the substitution is not done by the shell, it's just similar syntax (as mentioned in the doc http://logback.qos.ch/manual/configuration.html#variableSubstitution).
> Chapter 3: Logback configuration - QOS.ch<http://logback.qos.ch/manual/configuration.html#variableSubstitution>
> Thus, the level of the root logger has no apparent effect because the loggers in chapters.configuration.MyApp3 and chapters.configuration.Foo classes are all enabled for the INFO level. As a side note, the chapters.configuration logger exists by virtue of its declaration in the configuration file - even if the Java source code does not directly refer to it.
> logback.qos.ch
>
>
> Regards,
> Alexis.
>
> ________________________________
> From: houssem <me...@gmail.com>
> Sent: Wednesday, September 1, 2021 7:02 PM
> To: user@flink.apache.org <us...@flink.apache.org>
> Subject: Re: logback variable substitution in kubernetes
>
>
> Yes i did it all
>
> when i hard code the log level and the file name everything works fine
>
> but when i try to use variables , they won't be replaced.
>
> On 2021/09/01 11:43:21, Yang Wang <da...@gmail.com> wrote:
> > Did you have removed the log4j related jars in the $FLINK_HOME/lib
> > directory?
> > Refer to the documentation[1] for how to use logback.
> >
> > [1].
> > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
> >
> > Best,
> > Yang
> >
> > houssem <me...@gmail.com> 于2021年9月1日周三 下午5:00写道：
> >
> > > Yes i did this verification and i have all environment variables.
> > >
> > > On 2021/09/01 06:09:27, Yang Wang <da...@gmail.com> wrote:
> > > > From the logback documentation[1], it could support OS
> > > > environment substitution.
> > > > Could you please check that the environment variables have been properly
> > > > set?
> > > > Maybe you could tunnel into the Kubernetes pod via "kubectl exec" and do
> > > > such verification.
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > houssem <me...@gmail.com> 于2021年8月31日周二 下午7:28写道：
> > > >
> > > > >
> > > > > Hello,
> > > > >
> > > > > I am running a flink application cluster in standalone kubernetes mode
> > > and
> > > > > i a using logback
> > > > >
> > > > > as a logging framework , th problem is i am not able tu use environment
> > > > > variables configured in my pod inside my logback-console.xml file .
> > > > >
> > > > >
> > > > >
> > > > > I copied this file from my file system while building my image.
> > > > > Dockerfile:
> > > > > ..
> > > > > COPY logback-console.xml $FLINK_HOME/conf/
> > > > > ..
> > > > >
> > > > >
> > > > > here is my logback-console file::
> > > > >
> > > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > > <configuration>
> > > > >
> > > > > <logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
> > > > > <logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
> > > > > <logger name="akka" level="$FLINK_LOG_LEVEL" />
> > > > > <logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
> > > > > <logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
> > > > > ####Appenders####
> > > > > #file
> > > > > <appender name="file" class="ch.qos.logback.core.FileAppender">
> > > > > <file>$LOG_FILE</file>
> > > > > <encoder
> > > class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > > > <pattern>%n</pattern>
> > > > > </suffix>
> > > > > </layout>
> > > > > </encoder>
> > > > > </appender>
> > > > >
> > > > > #console
> > > > > <appender name="stdout"
> > > class="ch.qos.logback.core.ConsoleAppender">
> > > > > <!-- encoders are assigned the type
> > > > > ch.qos.logback.classic.encoder.PatternLayoutEncoder by
> > > > > default -->
> > > > > <encoder
> > > class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > > > <pattern>%n</pattern>
> > > > > </suffix>
> > > > > </layout>
> > > > > </encoder>
> > > > > </appender>
> > > > >
> > > > > <root level="$LOG_LEVEL">
> > > > > <appender-ref ref="file"/>
> > > > > <appender-ref ref="stdout"/>
> > > > > </root>
> > > > >
> > > >
> > >
> >
>
Re: logback variable substitution in kubernetes
Posted by Alexis Sarda-Espinosa <al...@microfocus.com>.
I'm fairly certain you need the curly braces surrounding the variable, the substitution is not done by the shell, it's just similar syntax (as mentioned in the doc http://logback.qos.ch/manual/configuration.html#variableSubstitution).
Chapter 3: Logback configuration - QOS.ch<http://logback.qos.ch/manual/configuration.html#variableSubstitution>
Thus, the level of the root logger has no apparent effect because the loggers in chapters.configuration.MyApp3 and chapters.configuration.Foo classes are all enabled for the INFO level. As a side note, the chapters.configuration logger exists by virtue of its declaration in the configuration file - even if the Java source code does not directly refer to it.
logback.qos.ch
Regards,
Alexis.
________________________________
From: houssem <me...@gmail.com>
Sent: Wednesday, September 1, 2021 7:02 PM
To: user@flink.apache.org <us...@flink.apache.org>
Subject: Re: logback variable substitution in kubernetes
Yes i did it all
when i hard code the log level and the file name everything works fine
but when i try to use variables , they won't be replaced.
On 2021/09/01 11:43:21, Yang Wang <da...@gmail.com> wrote:
> Did you have removed the log4j related jars in the $FLINK_HOME/lib
> directory?
> Refer to the documentation[1] for how to use logback.
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
>
> Best,
> Yang
>
> houssem <me...@gmail.com> 于2021年9月1日周三 下午5:00写道：
>
> > Yes i did this verification and i have all environment variables.
> >
> > On 2021/09/01 06:09:27, Yang Wang <da...@gmail.com> wrote:
> > > From the logback documentation[1], it could support OS
> > > environment substitution.
> > > Could you please check that the environment variables have been properly
> > > set?
> > > Maybe you could tunnel into the Kubernetes pod via "kubectl exec" and do
> > > such verification.
> > >
> > > Best,
> > > Yang
> > >
> > > houssem <me...@gmail.com> 于2021年8月31日周二 下午7:28写道：
> > >
> > > >
> > > > Hello,
> > > >
> > > > I am running a flink application cluster in standalone kubernetes mode
> > and
> > > > i a using logback
> > > >
> > > > as a logging framework , th problem is i am not able tu use environment
> > > > variables configured in my pod inside my logback-console.xml file .
> > > >
> > > >
> > > >
> > > > I copied this file from my file system while building my image.
> > > > Dockerfile:
> > > > ..
> > > > COPY logback-console.xml $FLINK_HOME/conf/
> > > > ..
> > > >
> > > >
> > > > here is my logback-console file::
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > <configuration>
> > > >
> > > > <logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
> > > > <logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
> > > > <logger name="akka" level="$FLINK_LOG_LEVEL" />
> > > > <logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
> > > > <logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
> > > > ####Appenders####
> > > > #file
> > > > <appender name="file" class="ch.qos.logback.core.FileAppender">
> > > > <file>$LOG_FILE</file>
> > > > <encoder
> > class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > > <pattern>%n</pattern>
> > > > </suffix>
> > > > </layout>
> > > > </encoder>
> > > > </appender>
> > > >
> > > > #console
> > > > <appender name="stdout"
> > class="ch.qos.logback.core.ConsoleAppender">
> > > > <!-- encoders are assigned the type
> > > > ch.qos.logback.classic.encoder.PatternLayoutEncoder by
> > > > default -->
> > > > <encoder
> > class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > > <pattern>%n</pattern>
> > > > </suffix>
> > > > </layout>
> > > > </encoder>
> > > > </appender>
> > > >
> > > > <root level="$LOG_LEVEL">
> > > > <appender-ref ref="file"/>
> > > > <appender-ref ref="stdout"/>
> > > > </root>
> > > >
> > >
> >
>
Re: logback variable substitution in kubernetes
Posted by houssem <me...@gmail.com>.
Yes i did it all
when i hard code the log level and the file name everything works fine
but when i try to use variables , they won't be replaced.
On 2021/09/01 11:43:21, Yang Wang <da...@gmail.com> wrote:
> Did you have removed the log4j related jars in the $FLINK_HOME/lib
> directory?
> Refer to the documentation[1] for how to use logback.
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
>
> Best,
> Yang
>
> houssem <me...@gmail.com> 于2021年9月1日周三 下午5:00写道：
>
> > Yes i did this verification and i have all environment variables.
> >
> > On 2021/09/01 06:09:27, Yang Wang <da...@gmail.com> wrote:
> > > From the logback documentation[1], it could support OS
> > > environment substitution.
> > > Could you please check that the environment variables have been properly
> > > set?
> > > Maybe you could tunnel into the Kubernetes pod via "kubectl exec" and do
> > > such verification.
> > >
> > > Best,
> > > Yang
> > >
> > > houssem <me...@gmail.com> 于2021年8月31日周二 下午7:28写道：
> > >
> > > >
> > > > Hello,
> > > >
> > > > I am running a flink application cluster in standalone kubernetes mode
> > and
> > > > i a using logback
> > > >
> > > > as a logging framework , th problem is i am not able tu use environment
> > > > variables configured in my pod inside my logback-console.xml file .
> > > >
> > > >
> > > >
> > > > I copied this file from my file system while building my image.
> > > > Dockerfile:
> > > > ..
> > > > COPY logback-console.xml $FLINK_HOME/conf/
> > > > ..
> > > >
> > > >
> > > > here is my logback-console file::
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"?>
> > > > <configuration>
> > > >
> > > > <logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
> > > > <logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
> > > > <logger name="akka" level="$FLINK_LOG_LEVEL" />
> > > > <logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
> > > > <logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
> > > > ####Appenders####
> > > > #file
> > > > <appender name="file" class="ch.qos.logback.core.FileAppender">
> > > > <file>$LOG_FILE</file>
> > > > <encoder
> > class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > > <pattern>%n</pattern>
> > > > </suffix>
> > > > </layout>
> > > > </encoder>
> > > > </appender>
> > > >
> > > > #console
> > > > <appender name="stdout"
> > class="ch.qos.logback.core.ConsoleAppender">
> > > > <!-- encoders are assigned the type
> > > > ch.qos.logback.classic.encoder.PatternLayoutEncoder by
> > > > default -->
> > > > <encoder
> > class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > > <pattern>%n</pattern>
> > > > </suffix>
> > > > </layout>
> > > > </encoder>
> > > > </appender>
> > > >
> > > > <root level="$LOG_LEVEL">
> > > > <appender-ref ref="file"/>
> > > > <appender-ref ref="stdout"/>
> > > > </root>
> > > >
> > >
> >
>
Re: logback variable substitution in kubernetes
Posted by Yang Wang <da...@gmail.com>.
Did you have removed the log4j related jars in the $FLINK_HOME/lib
directory?
Refer to the documentation[1] for how to use logback.
[1].
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
Best,
Yang
houssem <me...@gmail.com> 于2021年9月1日周三 下午5:00写道：
> Yes i did this verification and i have all environment variables.
>
> On 2021/09/01 06:09:27, Yang Wang <da...@gmail.com> wrote:
> > From the logback documentation[1], it could support OS
> > environment substitution.
> > Could you please check that the environment variables have been properly
> > set?
> > Maybe you could tunnel into the Kubernetes pod via "kubectl exec" and do
> > such verification.
> >
> > Best,
> > Yang
> >
> > houssem <me...@gmail.com> 于2021年8月31日周二 下午7:28写道：
> >
> > >
> > > Hello,
> > >
> > > I am running a flink application cluster in standalone kubernetes mode
> and
> > > i a using logback
> > >
> > > as a logging framework , th problem is i am not able tu use environment
> > > variables configured in my pod inside my logback-console.xml file .
> > >
> > >
> > >
> > > I copied this file from my file system while building my image.
> > > Dockerfile:
> > > ..
> > > COPY logback-console.xml $FLINK_HOME/conf/
> > > ..
> > >
> > >
> > > here is my logback-console file::
> > >
> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <configuration>
> > >
> > > <logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
> > > <logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
> > > <logger name="akka" level="$FLINK_LOG_LEVEL" />
> > > <logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
> > > <logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
> > > ####Appenders####
> > > #file
> > > <appender name="file" class="ch.qos.logback.core.FileAppender">
> > > <file>$LOG_FILE</file>
> > > <encoder
> class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > <pattern>%n</pattern>
> > > </suffix>
> > > </layout>
> > > </encoder>
> > > </appender>
> > >
> > > #console
> > > <appender name="stdout"
> class="ch.qos.logback.core.ConsoleAppender">
> > > <!-- encoders are assigned the type
> > > ch.qos.logback.classic.encoder.PatternLayoutEncoder by
> > > default -->
> > > <encoder
> class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > > <layout class="net.logstash.logback.layout.LogstashLayout">
> > > <suffix class="ch.qos.logback.classic.PatternLayout">
> > > <pattern>%n</pattern>
> > > </suffix>
> > > </layout>
> > > </encoder>
> > > </appender>
> > >
> > > <root level="$LOG_LEVEL">
> > > <appender-ref ref="file"/>
> > > <appender-ref ref="stdout"/>
> > > </root>
> > >
> >
>
Re: logback variable substitution in kubernetes
Posted by houssem <me...@gmail.com>.
Yes i did this verification and i have all environment variables.
On 2021/09/01 06:09:27, Yang Wang <da...@gmail.com> wrote:
> From the logback documentation[1], it could support OS
> environment substitution.
> Could you please check that the environment variables have been properly
> set?
> Maybe you could tunnel into the Kubernetes pod via "kubectl exec" and do
> such verification.
>
> Best,
> Yang
>
> houssem <me...@gmail.com> 于2021年8月31日周二 下午7:28写道：
>
> >
> > Hello,
> >
> > I am running a flink application cluster in standalone kubernetes mode and
> > i a using logback
> >
> > as a logging framework , th problem is i am not able tu use environment
> > variables configured in my pod inside my logback-console.xml file .
> >
> >
> >
> > I copied this file from my file system while building my image.
> > Dockerfile:
> > ..
> > COPY logback-console.xml $FLINK_HOME/conf/
> > ..
> >
> >
> > here is my logback-console file::
> >
> > <?xml version="1.0" encoding="UTF-8"?>
> > <configuration>
> >
> > <logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
> > <logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
> > <logger name="akka" level="$FLINK_LOG_LEVEL" />
> > <logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
> > <logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
> > ####Appenders####
> > #file
> > <appender name="file" class="ch.qos.logback.core.FileAppender">
> > <file>$LOG_FILE</file>
> > <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > <layout class="net.logstash.logback.layout.LogstashLayout">
> > <suffix class="ch.qos.logback.classic.PatternLayout">
> > <pattern>%n</pattern>
> > </suffix>
> > </layout>
> > </encoder>
> > </appender>
> >
> > #console
> > <appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
> > <!-- encoders are assigned the type
> > ch.qos.logback.classic.encoder.PatternLayoutEncoder by
> > default -->
> > <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> > <layout class="net.logstash.logback.layout.LogstashLayout">
> > <suffix class="ch.qos.logback.classic.PatternLayout">
> > <pattern>%n</pattern>
> > </suffix>
> > </layout>
> > </encoder>
> > </appender>
> >
> > <root level="$LOG_LEVEL">
> > <appender-ref ref="file"/>
> > <appender-ref ref="stdout"/>
> > </root>
> >
>
Re: logback variable substitution in kubernetes
Posted by Yang Wang <da...@gmail.com>.
From the logback documentation[1], it could support OS
environment substitution.
Could you please check that the environment variables have been properly
set?
Maybe you could tunnel into the Kubernetes pod via "kubectl exec" and do
such verification.
Best,
Yang
houssem <me...@gmail.com> 于2021年8月31日周二 下午7:28写道：
>
> Hello,
>
> I am running a flink application cluster in standalone kubernetes mode and
> i a using logback
>
> as a logging framework , th problem is i am not able tu use environment
> variables configured in my pod inside my logback-console.xml file .
>
>
>
> I copied this file from my file system while building my image.
> Dockerfile:
> ..
> COPY logback-console.xml $FLINK_HOME/conf/
> ..
>
>
> here is my logback-console file::
>
> <?xml version="1.0" encoding="UTF-8"?>
> <configuration>
>
> <logger name="com.cpm.sdf.notif" level="$LOG_LEVEL " />
> <logger name="org.cpm.flink" level="$FLINK_LOG_LEVEL" />
> <logger name="akka" level="$FLINK_LOG_LEVEL" />
> <logger name="org.apache.hadoop" level="$FLINK_LOG_LEVEL" />
> <logger name="org.apache.zookeeper" level="$FLINK_LOG_LEVEL" />
> ####Appenders####
> #file
> <appender name="file" class="ch.qos.logback.core.FileAppender">
> <file>$LOG_FILE</file>
> <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> <layout class="net.logstash.logback.layout.LogstashLayout">
> <suffix class="ch.qos.logback.classic.PatternLayout">
> <pattern>%n</pattern>
> </suffix>
> </layout>
> </encoder>
> </appender>
>
> #console
> <appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
> <!-- encoders are assigned the type
> ch.qos.logback.classic.encoder.PatternLayoutEncoder by
> default -->
> <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
> <layout class="net.logstash.logback.layout.LogstashLayout">
> <suffix class="ch.qos.logback.classic.PatternLayout">
> <pattern>%n</pattern>
> </suffix>
> </layout>
> </encoder>
> </appender>
>
> <root level="$LOG_LEVEL">
> <appender-ref ref="file"/>
> <appender-ref ref="stdout"/>
> </root>
>