You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Fuyao Li <fu...@oracle.com> on 2021/09/24 23:57:09 UTC
How to enable customize logging library based on SLF4J for Flink
deployment in Kubernetes
Hi Flink Community,
I am trying enable a company internal logging framework built upon SLF4J and log4j. This logging framework has another separate jar and specific logging configurations. After debugging, I am able to make Flink application running correctly in the local IDE with the internal logging framework after adding related SLF4J, log4j dependencies, and logging framework dependencies.
However, I still run into errors when I deploy this into the Kubernetes environment. I tried to add the logging framework jar to /opt/flink/lib/ folder, but it doesn’t help much. I am not sure which part I am missing here. I have attached relevant information below. Thanks for your help.
This is the log4j2-console.properties I proposed, I have injected this as a configmap (mounted to /opt/flink/conf inside the pod using a Flink native Kubernetes Operator I build).
Such configuration will run correctly in Local IDE and generate logs in the internal logging framework expected shape. (I have rename it to log4j2.properties and put it into resources/ folder during local debug.)
packages = oracle.spectra.logging.base
status = WARN
monitorInterval = 30
shutdownHook = disable
rootLogger.level = ${sys:spectra-log-level:-INFO}
rootLogger.appenderRef.asyncC.ref = AsyncCAppender
rootLogger.appenderRef.asyncF.ref = AsyncFAppender
appender.asyncC.name = AsyncCAppender
appender.asyncC.type = Async
appender.asyncC.bufferSize = 256
appender.asyncC.appenderRef.type = AppenderRef
appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
# Log all infos to the console
appender.console.name = JSONLogConsoleAppender
appender.console.target = SYSTEM_OUT
appender.console.type = Console
appender.console.layout.type = SpectraJsonLayout
appender.console.layout.compact = true
appender.console.layout.eventEol = true
appender.asyncF.name = AsyncFAppender
appender.asyncF.type = Async
appender.asyncF.bufferSize = 256
appender.asyncF.appenderRef.type = AppenderRef
appender.asyncF.appenderRef.ref = RollingFileAppender
# Log all infos in the given rolling file
appender.rolling.type = RollingFile
appender.rolling.name = RollingFileAppender
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = SpectraJsonLayout
appender.rolling.layout.compact = false
appender.rolling.layout.eventEol = true
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
This is the error I got from the Job Manager pod in the Kubernetes.
sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only file system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only file system
/docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml: Read-only file system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting kubernetes-application as a console application on host faw-poc-demo-67864b696b-9zbbc.
2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for SpectraJsonLayout
2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for SpectraJsonLayout
2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,689 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,758 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,766 main ERROR Null object returned for RollingFile in Appenders.
2021-09-24 23:42:53,766 main ERROR Null object returned for Console in Appenders.
2021-09-24 23:42:53,771 main ERROR No appender named RollingFileAppender was configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No appenders are available for AsyncAppender AsyncFAppender
at org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
As you can see, the SpectraJsonLayout that is defined in the logging framework jar is somehow not recognized. I am not sure how to solve this? Could anyone share some information. Really appreciate your help!
Best regards,
Fuyao
Re: [External] : Re: How to enable customize logging library based on
SLF4J for Flink deployment in Kubernetes
Posted by Fuyao Li <fu...@oracle.com>.
Thanks! I got your point. Will try it out.
From: Chesnay Schepler <ch...@apache.org>
Date: Tuesday, October 19, 2021 at 01:44
To: Fuyao Li <fu...@oracle.com>, user <us...@flink.apache.org>
Cc: Rohit Gupta <ro...@oracle.com>
Subject: Re: [External] : Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes
1) Adding it as a dependency to the Flink application does not work with an actual Flink cluster, because said dependency must be available when the cluster is started. It works in the IDE because there everything is put onto the same classpath.
2) folder structure shouldn't be relevant. So long as it is in somewhere in the lib directory of the Flink distribution it should be picked up.
3) Put the jar into the lib directory, do not add the config map, start the cluster and check the logs. The classpath will be logged very early in the startup procedure.
On 18/10/2021 22:52, Fuyao Li wrote:
Hi Chesnay,
Thanks for the reply.
1. The internal logging framework is built upon slf4j/log4j2 (The same one Flink uses, but it comes with an additional POM dependency). I have added such dependency in the Flink application POM file. But it seems only to work locally in IDE. When it is in the Flink cluster environment, it can’t work.
2. I tried to only add the configmap and put a single jar into lib/ folder, and it seems it still can’t find the classpath. How should I organize the folder structure? /lib/internal-logging/xxx.jar or this jar file must be directly under /lib, something like /lib/xxx.jar?
3. I got you point, I guess it is stilling using Flink default logging classpath and that causes the issue of not recognizing the internal framework? How to check the classpath of the Flink logging? Could you share me some blogs..? I am not familiar with this.
Best,
Fuyao
From: Chesnay Schepler <ch...@apache.org>
Date: Tuesday, September 28, 2021 at 07:06
To: Fuyao Li <fu...@oracle.com>, user <us...@flink.apache.org>
Cc: Rohit Gupta <ro...@oracle.com>
Subject: [External] : Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes
Could you clarify whether this internal framework uses a custom slfj4/log4j2 version, or is it just using what Flink comes with?
Did you only add the configmap and put a single jar into lib, or did you make other changes in Flink?
Can you remove just the configmap, start the cluster, and provide us with the classpath that Flink is logging?
On 25/09/2021 01:57, Fuyao Li wrote:
Hi Flink Community,
I am trying enable a company internal logging framework built upon SLF4J and log4j. This logging framework has another separate jar and specific logging configurations. After debugging, I am able to make Flink application running correctly in the local IDE with the internal logging framework after adding related SLF4J, log4j dependencies, and logging framework dependencies.
However, I still run into errors when I deploy this into the Kubernetes environment. I tried to add the logging framework jar to /opt/flink/lib/ folder, but it doesn’t help much. I am not sure which part I am missing here. I have attached relevant information below. Thanks for your help.
This is the log4j2-console.properties I proposed, I have injected this as a configmap (mounted to /opt/flink/conf inside the pod using a Flink native Kubernetes Operator I build).
Such configuration will run correctly in Local IDE and generate logs in the internal logging framework expected shape. (I have rename it to log4j2.properties and put it into resources/ folder during local debug.)
packages = oracle.spectra.logging.base
status = WARN
monitorInterval = 30
shutdownHook = disable
rootLogger.level = ${sys:spectra-log-level:-INFO}
rootLogger.appenderRef.asyncC.ref = AsyncCAppender
rootLogger.appenderRef.asyncF.ref = AsyncFAppender
appender.asyncC.name = AsyncCAppender
appender.asyncC.type = Async
appender.asyncC.bufferSize = 256
appender.asyncC.appenderRef.type = AppenderRef
appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
# Log all infos to the console
appender.console.name = JSONLogConsoleAppender
appender.console.target = SYSTEM_OUT
appender.console.type = Console
appender.console.layout.type = SpectraJsonLayout
appender.console.layout.compact = true
appender.console.layout.eventEol = true
appender.asyncF.name = AsyncFAppender
appender.asyncF.type = Async
appender.asyncF.bufferSize = 256
appender.asyncF.appenderRef.type = AppenderRef
appender.asyncF.appenderRef.ref = RollingFileAppender
# Log all infos in the given rolling file
appender.rolling.type = RollingFile
appender.rolling.name = RollingFileAppender
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = SpectraJsonLayout
appender.rolling.layout.compact = false
appender.rolling.layout.eventEol = true
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
This is the error I got from the Job Manager pod in the Kubernetes.
sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only file system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only file system
/docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml: Read-only file system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting kubernetes-application as a console application on host faw-poc-demo-67864b696b-9zbbc.
2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for SpectraJsonLayout
2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for SpectraJsonLayout
2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,689 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,758 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,766 main ERROR Null object returned for RollingFile in Appenders.
2021-09-24 23:42:53,766 main ERROR Null object returned for Console in Appenders.
2021-09-24 23:42:53,771 main ERROR No appender named RollingFileAppender was configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No appenders are available for AsyncAppender AsyncFAppender
at org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
As you can see, the SpectraJsonLayout that is defined in the logging framework jar is somehow not recognized. I am not sure how to solve this? Could anyone share some information. Really appreciate your help!
Best regards,
Fuyao
Re: [External] : Re: How to enable customize logging library based on
SLF4J for Flink deployment in Kubernetes
Posted by Chesnay Schepler <ch...@apache.org>.
1) Adding it as a dependency to the Flink application does not work with
an actual Flink cluster, because said dependency must be available when
the cluster is /started/. It works in the IDE because there everything
is put onto the same classpath.
2) folder structure shouldn't be relevant. So long as it is in somewhere
in the lib directory of the Flink distribution it should be picked up.
3) Put the jar into the lib directory, do /not/ add the config map,
start the cluster and check the logs. The classpath will be logged very
early in the startup procedure.
On 18/10/2021 22:52, Fuyao Li wrote:
>
> Hi Chesnay,
>
> Thanks for the reply.
>
> 1. The internal logging framework is built upon slf4j/log4j2 (The
> same one Flink uses, but it comes with an additional POM
> dependency). I have added such dependency in the Flink application
> POM file. But it seems only to work locally in IDE. When it is in
> the Flink cluster environment, it can’t work.
> 2. I tried to only add the configmap and put a single jar into lib/
> folder, and it seems it still can’t find the classpath. How should
> I organize the folder structure? /lib/internal-logging/xxx.jar or
> this jar file must be directly under /lib, something like
> /lib/xxx.jar?
> 3. I got you point, I guess it is stilling using Flink default
> logging classpath and that causes the issue of not recognizing the
> internal framework? How to check the classpath of the Flink
> logging? Could you share me some blogs..? I am not familiar with this.
>
> Best,
>
> Fuyao
>
> *From: *Chesnay Schepler <ch...@apache.org>
> *Date: *Tuesday, September 28, 2021 at 07:06
> *To: *Fuyao Li <fu...@oracle.com>, user <us...@flink.apache.org>
> *Cc: *Rohit Gupta <ro...@oracle.com>
> *Subject: *[External] : Re: How to enable customize logging library
> based on SLF4J for Flink deployment in Kubernetes
>
> Could you clarify whether this internal framework uses a custom
> slfj4/log4j2 version, or is it just using what Flink comes with?
>
> Did you only add the configmap and put a single jar into lib, or did
> you make other changes in Flink?
>
> Can you remove just the configmap, start the cluster, and provide us
> with the classpath that Flink is logging?
>
> On 25/09/2021 01:57, Fuyao Li wrote:
>
> Hi Flink Community,
>
> I am trying enable a company internal logging framework built upon
> SLF4J and log4j. This logging framework has another separate jar
> and specific logging configurations. After debugging, I am able to
> make Flink application running correctly in the local IDE with the
> internal logging framework after adding related SLF4J, log4j
> dependencies, and logging framework dependencies.
>
> However, I still run into errors when I deploy this into the
> Kubernetes environment. I tried to add the logging framework jar
> to /opt/flink/lib/ folder, but it doesn’t help much. I am not sure
> which part I am missing here. I have attached relevant information
> below. Thanks for your help.
>
> This is the log4j2-console.properties I proposed, I have injected
> this as a configmap (mounted to /opt/flink/conf inside the pod
> using a Flink native Kubernetes Operator I build).
>
> Such configuration will run correctly in Local IDE and generate
> logs in the internal logging framework expected shape. (I have
> rename it to log4j2.properties and put it into resources/ folder
> during local debug.)
>
> packages = oracle.spectra.logging.base
>
> status = WARN
>
> monitorInterval = 30
>
> shutdownHook = disable
>
> rootLogger.level = ${sys:spectra-log-level:-INFO}
>
> rootLogger.appenderRef.asyncC.ref = AsyncCAppender
>
> rootLogger.appenderRef.asyncF.ref = AsyncFAppender
>
> appender.asyncC.name = AsyncCAppender
>
> appender.asyncC.type = Async
>
> appender.asyncC.bufferSize = 256
>
> appender.asyncC.appenderRef.type = AppenderRef
>
> appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
>
> # Log all infos to the console
>
> appender.console.name = JSONLogConsoleAppender
>
> appender.console.target = SYSTEM_OUT
>
> appender.console.type = Console
>
> appender.console.layout.type = SpectraJsonLayout
>
> appender.console.layout.compact = *true*
>
> appender.console.layout.eventEol = *true*
>
> appender.asyncF.name = AsyncFAppender
>
> appender.asyncF.type = Async
>
> appender.asyncF.bufferSize = 256
>
> appender.asyncF.appenderRef.type = AppenderRef
>
> appender.asyncF.appenderRef.ref = RollingFileAppender
>
> # Log all infos in the given rolling file
>
> appender.rolling.type = RollingFile
>
> appender.rolling.name = RollingFileAppender
>
> appender.rolling.fileName = ${sys:log.file}
>
> appender.rolling.filePattern = ${sys:log.file}.%i
>
> appender.rolling.layout.type = SpectraJsonLayout
>
> appender.rolling.layout.compact = *false*
>
> appender.rolling.layout.eventEol = *true*
>
> appender.rolling.policies.type = Policies
>
> appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
>
> appender.rolling.policies.size.size=100MB
>
> appender.rolling.strategy.type = DefaultRolloverStrategy
>
> appender.rolling.strategy.max = 10
>
> # Uncomment *this**if*you want to _only_ change Flink's logging
>
> #logger.flink.name = org.apache.flink
>
> #logger.flink.level = INFO
>
> # The following lines keep the log level of common
> libraries/connectors on
>
> # log level INFO. The root logger does not override *this*. You
> have to manually
>
> # change the log levels here.
>
> logger.akka.name = akka
>
> logger.akka.level = INFO
>
> logger.kafka.name= org.apache.kafka
>
> logger.kafka.level = INFO
>
> logger.hadoop.name = org.apache.hadoop
>
> logger.hadoop.level = INFO
>
> logger.zookeeper.name = org.apache.zookeeper
>
> logger.zookeeper.level = INFO
>
> # Suppress the irrelevant (wrong) warnings from the Netty channel
> handler
>
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
>
> logger.netty.level = OFF
>
> This is the error I got from the Job Manager pod in the Kubernetes.
>
> sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl:
> Read-only file system
>
> sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o:
> Read-only file system
>
> /docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml:
> Read-only file system
>
> sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn:
> Read-only file system
>
> /docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml:
> Read-only file system
>
> /docker-entrypoint.sh: line 88:
> /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
>
> Starting kubernetes-application as a console application on host
> faw-poc-demo-67864b696b-9zbbc.
>
> 2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type
> for SpectraJsonLayout
>
> 2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type
> for SpectraJsonLayout
>
> 2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for
> SpectraJsonLayout
>
> 2021-09-24 23:42:53,689 main ERROR Could not create plugin of type
> class org.apache.logging.log4j.core.appender.RollingFileAppender
> for element RollingFile: java.lang.NullPointerException
> java.lang.NullPointerException
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method
> in class
> org.apache.logging.log4j.core.appender.RollingFileAppender for
> element RollingFile: java.lang.IllegalStateException: No factory
> method found for class
> org.apache.logging.log4j.core.appender.RollingFileAppender
> java.lang.IllegalStateException: No factory method found for class
> org.apache.logging.log4j.core.appender.RollingFileAppender
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for
> SpectraJsonLayout
>
> 2021-09-24 23:42:53,758 main ERROR Could not create plugin of type
> class org.apache.logging.log4j.core.appender.ConsoleAppender for
> element Console: java.lang.NullPointerException
> java.lang.NullPointerException
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method
> in class org.apache.logging.log4j.core.appender.ConsoleAppender
> for element Console: java.lang.IllegalStateException: No factory
> method found for class
> org.apache.logging.log4j.core.appender.ConsoleAppender
> java.lang.IllegalStateException: No factory method found for class
> org.apache.logging.log4j.core.appender.ConsoleAppender
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,766 main ERROR Null object returned for
> RollingFile in Appenders.
>
> 2021-09-24 23:42:53,766 main ERROR Null object returned for
> Console in Appenders.
>
> 2021-09-24 23:42:53,771 main ERROR No appender named
> RollingFileAppender was configured
>
> Exception in thread "main" java.lang.ExceptionInInitializerError
>
> Caused by:
> org.apache.logging.log4j.core.config.ConfigurationException: No
> appenders are available for AsyncAppender AsyncFAppender
>
> at
> org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> As you can see, the SpectraJsonLayout that is defined in the
> logging framework jar is somehow not recognized. I am not sure how
> to solve this? Could anyone share some information. Really
> appreciate your help!
>
> Best regards,
>
> Fuyao
>
Re: [External] : Re: How to enable customize logging library based on
SLF4J for Flink deployment in Kubernetes
Posted by Fuyao Li <fu...@oracle.com>.
Hi Chesnay,
Thanks for the reply.
1. The internal logging framework is built upon slf4j/log4j2 (The same one Flink uses, but it comes with an additional POM dependency). I have added such dependency in the Flink application POM file. But it seems only to work locally in IDE. When it is in the Flink cluster environment, it can’t work.
2. I tried to only add the configmap and put a single jar into lib/ folder, and it seems it still can’t find the classpath. How should I organize the folder structure? /lib/internal-logging/xxx.jar or this jar file must be directly under /lib, something like /lib/xxx.jar?
3. I got you point, I guess it is stilling using Flink default logging classpath and that causes the issue of not recognizing the internal framework? How to check the classpath of the Flink logging? Could you share me some blogs..? I am not familiar with this.
Best,
Fuyao
From: Chesnay Schepler <ch...@apache.org>
Date: Tuesday, September 28, 2021 at 07:06
To: Fuyao Li <fu...@oracle.com>, user <us...@flink.apache.org>
Cc: Rohit Gupta <ro...@oracle.com>
Subject: [External] : Re: How to enable customize logging library based on SLF4J for Flink deployment in Kubernetes
Could you clarify whether this internal framework uses a custom slfj4/log4j2 version, or is it just using what Flink comes with?
Did you only add the configmap and put a single jar into lib, or did you make other changes in Flink?
Can you remove just the configmap, start the cluster, and provide us with the classpath that Flink is logging?
On 25/09/2021 01:57, Fuyao Li wrote:
Hi Flink Community,
I am trying enable a company internal logging framework built upon SLF4J and log4j. This logging framework has another separate jar and specific logging configurations. After debugging, I am able to make Flink application running correctly in the local IDE with the internal logging framework after adding related SLF4J, log4j dependencies, and logging framework dependencies.
However, I still run into errors when I deploy this into the Kubernetes environment. I tried to add the logging framework jar to /opt/flink/lib/ folder, but it doesn’t help much. I am not sure which part I am missing here. I have attached relevant information below. Thanks for your help.
This is the log4j2-console.properties I proposed, I have injected this as a configmap (mounted to /opt/flink/conf inside the pod using a Flink native Kubernetes Operator I build).
Such configuration will run correctly in Local IDE and generate logs in the internal logging framework expected shape. (I have rename it to log4j2.properties and put it into resources/ folder during local debug.)
packages = oracle.spectra.logging.base
status = WARN
monitorInterval = 30
shutdownHook = disable
rootLogger.level = ${sys:spectra-log-level:-INFO}
rootLogger.appenderRef.asyncC.ref = AsyncCAppender
rootLogger.appenderRef.asyncF.ref = AsyncFAppender
appender.asyncC.name = AsyncCAppender
appender.asyncC.type = Async
appender.asyncC.bufferSize = 256
appender.asyncC.appenderRef.type = AppenderRef
appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
# Log all infos to the console
appender.console.name = JSONLogConsoleAppender
appender.console.target = SYSTEM_OUT
appender.console.type = Console
appender.console.layout.type = SpectraJsonLayout
appender.console.layout.compact = true
appender.console.layout.eventEol = true
appender.asyncF.name = AsyncFAppender
appender.asyncF.type = Async
appender.asyncF.bufferSize = 256
appender.asyncF.appenderRef.type = AppenderRef
appender.asyncF.appenderRef.ref = RollingFileAppender
# Log all infos in the given rolling file
appender.rolling.type = RollingFile
appender.rolling.name = RollingFileAppender
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = SpectraJsonLayout
appender.rolling.layout.compact = false
appender.rolling.layout.eventEol = true
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
This is the error I got from the Job Manager pod in the Kubernetes.
sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only file system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only file system
/docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml: Read-only file system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting kubernetes-application as a console application on host faw-poc-demo-67864b696b-9zbbc.
2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for SpectraJsonLayout
2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for SpectraJsonLayout
2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,689 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.RollingFileAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,758 main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.NullPointerException java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.appender.ConsoleAppender for element Console: java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender java.lang.IllegalStateException: No factory method found for class org.apache.logging.log4j.core.appender.ConsoleAppender
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,766 main ERROR Null object returned for RollingFile in Appenders.
2021-09-24 23:42:53,766 main ERROR Null object returned for Console in Appenders.
2021-09-24 23:42:53,771 main ERROR No appender named RollingFileAppender was configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No appenders are available for AsyncAppender AsyncFAppender
at org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
As you can see, the SpectraJsonLayout that is defined in the logging framework jar is somehow not recognized. I am not sure how to solve this? Could anyone share some information. Really appreciate your help!
Best regards,
Fuyao
Re: How to enable customize logging library based on SLF4J for Flink
deployment in Kubernetes
Posted by Chesnay Schepler <ch...@apache.org>.
Could you clarify whether this internal framework uses a custom
slfj4/log4j2 version, or is it just using what Flink comes with?
Did you only add the configmap and put a single jar into lib, or did you
make other changes in Flink?
Can you remove just the configmap, start the cluster, and provide us
with the classpath that Flink is logging?
On 25/09/2021 01:57, Fuyao Li wrote:
>
> Hi Flink Community,
>
> I am trying enable a company internal logging framework built upon
> SLF4J and log4j. This logging framework has another separate jar and
> specific logging configurations. After debugging, I am able to make
> Flink application running correctly in the local IDE with the internal
> logging framework after adding related SLF4J, log4j dependencies, and
> logging framework dependencies.
>
> However, I still run into errors when I deploy this into the
> Kubernetes environment. I tried to add the logging framework jar to
> /opt/flink/lib/ folder, but it doesn’t help much. I am not sure which
> part I am missing here. I have attached relevant information below.
> Thanks for your help.
>
> This is the log4j2-console.properties I proposed, I have injected this
> as a configmap (mounted to /opt/flink/conf inside the pod using a
> Flink native Kubernetes Operator I build).
>
> Such configuration will run correctly in Local IDE and generate logs
> in the internal logging framework expected shape. (I have rename it to
> log4j2.properties and put it into resources/ folder during local debug.)
>
> packages = oracle.spectra.logging.base
>
> status = WARN
>
> monitorInterval = 30
>
> shutdownHook = disable
>
> rootLogger.level = ${sys:spectra-log-level:-INFO}
>
> rootLogger.appenderRef.asyncC.ref = AsyncCAppender
>
> rootLogger.appenderRef.asyncF.ref = AsyncFAppender
>
> appender.asyncC.name = AsyncCAppender
>
> appender.asyncC.type = Async
>
> appender.asyncC.bufferSize = 256
>
> appender.asyncC.appenderRef.type = AppenderRef
>
> appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
>
> # Log all infos to the console
>
> appender.console.name = JSONLogConsoleAppender
>
> appender.console.target = SYSTEM_OUT
>
> appender.console.type = Console
>
> appender.console.layout.type = SpectraJsonLayout
>
> appender.console.layout.compact = *true*
>
> appender.console.layout.eventEol = *true*
>
> appender.asyncF.name = AsyncFAppender
>
> appender.asyncF.type = Async
>
> appender.asyncF.bufferSize = 256
>
> appender.asyncF.appenderRef.type = AppenderRef
>
> appender.asyncF.appenderRef.ref = RollingFileAppender
>
> # Log all infos in the given rolling file
>
> appender.rolling.type = RollingFile
>
> appender.rolling.name = RollingFileAppender
>
> appender.rolling.fileName = ${sys:log.file}
>
> appender.rolling.filePattern = ${sys:log.file}.%i
>
> appender.rolling.layout.type = SpectraJsonLayout
>
> appender.rolling.layout.compact = *false*
>
> appender.rolling.layout.eventEol = *true*
>
> appender.rolling.policies.type = Policies
>
> appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
>
> appender.rolling.policies.size.size=100MB
>
> appender.rolling.strategy.type = DefaultRolloverStrategy
>
> appender.rolling.strategy.max = 10
>
> # Uncomment *this**if*you want to _only_ change Flink's logging
>
> #logger.flink.name = org.apache.flink
>
> #logger.flink.level = INFO
>
> # The following lines keep the log level of common libraries/connectors on
>
> # log level INFO. The root logger does not override *this*. You have
> to manually
>
> # change the log levels here.
>
> logger.akka.name = akka
>
> logger.akka.level = INFO
>
> logger.kafka.name= org.apache.kafka
>
> logger.kafka.level = INFO
>
> logger.hadoop.name = org.apache.hadoop
>
> logger.hadoop.level = INFO
>
> logger.zookeeper.name = org.apache.zookeeper
>
> logger.zookeeper.level = INFO
>
> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
>
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
>
> logger.netty.level = OFF
>
> This is the error I got from the Job Manager pod in the Kubernetes.
>
> sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only
> file system
>
> sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only
> file system
>
> /docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml:
> Read-only file system
>
> sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only
> file system
>
> /docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml:
> Read-only file system
>
> /docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp:
> Read-only file system
>
> Starting kubernetes-application as a console application on host
> faw-poc-demo-67864b696b-9zbbc.
>
> 2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for
> SpectraJsonLayout
>
> 2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for
> SpectraJsonLayout
>
> 2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for
> SpectraJsonLayout
>
> 2021-09-24 23:42:53,689 main ERROR Could not create plugin of type
> class org.apache.logging.log4j.core.appender.RollingFileAppender for
> element RollingFile: java.lang.NullPointerException
> java.lang.NullPointerException
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in
> class org.apache.logging.log4j.core.appender.RollingFileAppender for
> element RollingFile: java.lang.IllegalStateException: No factory
> method found for class
> org.apache.logging.log4j.core.appender.RollingFileAppender
> java.lang.IllegalStateException: No factory method found for class
> org.apache.logging.log4j.core.appender.RollingFileAppender
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for
> SpectraJsonLayout
>
> 2021-09-24 23:42:53,758 main ERROR Could not create plugin of type
> class org.apache.logging.log4j.core.appender.ConsoleAppender for
> element Console: java.lang.NullPointerException
> java.lang.NullPointerException
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
>
> at
> org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in
> class org.apache.logging.log4j.core.appender.ConsoleAppender for
> element Console: java.lang.IllegalStateException: No factory method
> found for class org.apache.logging.log4j.core.appender.ConsoleAppender
> java.lang.IllegalStateException: No factory method found for class
> org.apache.logging.log4j.core.appender.ConsoleAppender
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
>
> at
> org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> 2021-09-24 23:42:53,766 main ERROR Null object returned for
> RollingFile in Appenders.
>
> 2021-09-24 23:42:53,766 main ERROR Null object returned for Console in
> Appenders.
>
> 2021-09-24 23:42:53,771 main ERROR No appender named
> RollingFileAppender was configured
>
> Exception in thread "main" java.lang.ExceptionInInitializerError
>
> Caused by:
> org.apache.logging.log4j.core.config.ConfigurationException: No
> appenders are available for AsyncAppender AsyncFAppender
>
> at
> org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
>
> at
> org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
>
> at
> org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
>
> at
> org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
>
> at
> org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
>
> at
> org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
>
> at
> org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
>
> at
> org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
>
> at
> org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>
> at
> org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
>
> As you can see, the SpectraJsonLayout that is defined in the logging
> framework jar is somehow not recognized. I am not sure how to solve
> this? Could anyone share some information. Really appreciate your help!
>
> Best regards,
>
> Fuyao
>