You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2019/07/31 13:21:00 UTC

Custom log appender for YARN

Hi All!

We are trying to configure a custom Kafka log appender for our YARN
application and we hit the following problem.

We included the log appender dependency in the fatjar of the application
because in YARN that should be part of the system class path.

However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint,
YarnTaskExecutorRunner) these only seem to have the contents of lib on
their classpath. Does someone have any pointers to how the classpath is
configured for running the entrypoints?

The exception is this:

java.lang.ClassNotFoundException:
org.apache.kafka.log4jappender.KafkaLog4jAppender
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)


Thanks,
Gyula

Re: Custom log appender for YARN

Posted by Biao Liu <mm...@gmail.com>.

Hi Gyula,

How about putting the appender class in lib folder, but choosing the
appender only for user code through log4j.properties? That way only the
user logger would be sent to Kafka. It seems what you want...

Thanks,
Biao /'bɪ.aʊ/



On Thu, Aug 1, 2019 at 2:25 PM Gyula Fóra <gy...@gmail.com> wrote:

> Hi Biao,
>
> Thanks for the input!
> This is basically where we got ourselves as well.
>
> We are trying to avoid putting things in the lib folder so separating the
> loggers would be great but we don't have a solution for it yet.
>
> Thanks
> Gyula
>
> On Thu, 1 Aug 2019, 07:09 Biao Liu <mmyy1110@gmail.com wrote:
>
>> Hi Gyula,
>>
>> I guess it should work if you put log appender jar under $flink/lib
>> folder.
>>
>> There are two different kinds of classloader in your case. One is for
>> Flink framework, the other is for user code. The framework classloader is
>> the parent of user classloader. The parent classloader could not find class
>> from sub-classloader due to the delegation model [1]. In your case, the `
>> ClusterEntrypoint` is in parent classloader, the log appender class in
>> fatjar is in sub-classloader.
>>
>> So I think there are two ways to solve this.
>> 1. Put your appender class under $flink/lib folder.
>> 2. Try to avoid using user-defined appender for Flink framework class.
>> (I'm not sure it could be done or not)
>>
>> You could find more informations in document [2] about Flink classloading
>> strategy.
>>
>> 1. https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html
>> 2.
>> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html
>>
>>
>> Thanks,
>> Biao /'bɪ.aʊ/
>>
>>
>>
>> On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <gy...@gmail.com> wrote:
>>
>>> Hi All!
>>>
>>> We are trying to configure a custom Kafka log appender for our YARN
>>> application and we hit the following problem.
>>>
>>> We included the log appender dependency in the fatjar of the application
>>> because in YARN that should be part of the system class path.
>>>
>>> However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint,
>>> YarnTaskExecutorRunner) these only seem to have the contents of lib on
>>> their classpath. Does someone have any pointers to how the classpath is
>>> configured for running the entrypoints?
>>>
>>> The exception is this:
>>>
>>> java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
>>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> 	at java.lang.Class.forName0(Native Method)
>>> 	at java.lang.Class.forName(Class.java:264)
>>> 	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
>>> 	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
>>> 	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
>>> 	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
>>> 	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
>>> 	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
>>> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
>>> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
>>> 	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
>>> 	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
>>> 	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
>>> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>>> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>>> 	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)
>>>
>>>
>>> Thanks,
>>> Gyula
>>>
>>>
>>>

Re: Custom log appender for YARN

Posted by Gyula Fóra <gy...@gmail.com>.

Hi Biao,

Thanks for the input!
This is basically where we got ourselves as well.

We are trying to avoid putting things in the lib folder so separating the
loggers would be great but we don't have a solution for it yet.

Thanks
Gyula

On Thu, 1 Aug 2019, 07:09 Biao Liu <mmyy1110@gmail.com wrote:

> Hi Gyula,
>
> I guess it should work if you put log appender jar under $flink/lib folder.
>
> There are two different kinds of classloader in your case. One is for
> Flink framework, the other is for user code. The framework classloader is
> the parent of user classloader. The parent classloader could not find class
> from sub-classloader due to the delegation model [1]. In your case, the `
> ClusterEntrypoint` is in parent classloader, the log appender class in
> fatjar is in sub-classloader.
>
> So I think there are two ways to solve this.
> 1. Put your appender class under $flink/lib folder.
> 2. Try to avoid using user-defined appender for Flink framework class.
> (I'm not sure it could be done or not)
>
> You could find more informations in document [2] about Flink classloading
> strategy.
>
> 1. https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html
> 2.
> https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html
>
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <gy...@gmail.com> wrote:
>
>> Hi All!
>>
>> We are trying to configure a custom Kafka log appender for our YARN
>> application and we hit the following problem.
>>
>> We included the log appender dependency in the fatjar of the application
>> because in YARN that should be part of the system class path.
>>
>> However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint,
>> YarnTaskExecutorRunner) these only seem to have the contents of lib on
>> their classpath. Does someone have any pointers to how the classpath is
>> configured for running the entrypoints?
>>
>> The exception is this:
>>
>> java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
>> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> 	at java.lang.Class.forName0(Native Method)
>> 	at java.lang.Class.forName(Class.java:264)
>> 	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
>> 	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
>> 	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
>> 	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
>> 	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
>> 	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
>> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
>> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
>> 	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
>> 	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
>> 	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
>> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
>> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
>> 	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)
>>
>>
>> Thanks,
>> Gyula
>>
>>
>>

Re: Custom log appender for YARN

Posted by Biao Liu <mm...@gmail.com>.

Hi Gyula,

I guess it should work if you put log appender jar under $flink/lib folder.

There are two different kinds of classloader in your case. One is for Flink
framework, the other is for user code. The framework classloader is the
parent of user classloader. The parent classloader could not find class
from sub-classloader due to the delegation model [1]. In your case, the `
ClusterEntrypoint` is in parent classloader, the log appender class in
fatjar is in sub-classloader.

So I think there are two ways to solve this.
1. Put your appender class under $flink/lib folder.
2. Try to avoid using user-defined appender for Flink framework class. (I'm
not sure it could be done or not)

You could find more informations in document [2] about Flink classloading
strategy.

1. https://docs.oracle.com/javase/8/docs/api/java/lang/ClassLoader.html
2.
https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html


Thanks,
Biao /'bɪ.aʊ/



On Wed, Jul 31, 2019 at 9:21 PM Gyula Fóra <gy...@gmail.com> wrote:

> Hi All!
>
> We are trying to configure a custom Kafka log appender for our YARN
> application and we hit the following problem.
>
> We included the log appender dependency in the fatjar of the application
> because in YARN that should be part of the system class path.
>
> However when the YARN cluster entrypoints run (YarnJobClusterEntrypoint,
> YarnTaskExecutorRunner) these only seem to have the contents of lib on
> their classpath. Does someone have any pointers to how the classpath is
> configured for running the entrypoints?
>
> The exception is this:
>
> java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
> 	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> 	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> 	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 	at java.lang.Class.forName0(Native Method)
> 	at java.lang.Class.forName(Class.java:264)
> 	at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
> 	at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
> 	at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
> 	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
> 	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
> 	at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
> 	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
> 	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> 	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
> 	at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:81)
> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
> 	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
> 	at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:104)
>
>
> Thanks,
> Gyula
>
>
>