You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2018/07/11 12:59:16 UTC

Custom metrics reporter classloading problem

Hi all,

I have ran into the following problem and I want to double check wether
this is intended behaviour.

I have a custom metrics reporter that pushes things to Kafka (so it creates
a KafkaProducer in the open method etc.etc.) for my streaming job.

Naturally as my Flink job consumes from Kafka so it has the kafka connector
dependencies I set the Kafka dependencies to provided in my metric reporter
project and I put the built kafkaReporter.jar into the Flink lib. However
it seems that the metrics reporter is instantiated without the user code
classes since I get a NoClassdefFound error for KafkaProducer even though
my streaming job starts successfully reading/writing kafka.

Any ideas why this happens and how to solve it? I am slightly against
putting the kafka dependencies twice on the classpath as it has only caused
problems in the past...

Gyula

Re: Custom metrics reporter classloading problem

Posted by Gyula Fóra <gy...@gmail.com>.
Thanks for the explanation, that makes sense.
For some reason I thought that in Yarn all stuff goes into the classpath.

Gy

Chesnay Schepler <ch...@apache.org> ezt írta (időpont: 2018. júl. 11.,
Sze, 15:16):

> Reporters do not have access to libraries provided with user-jars.
> They are instantiated when JM/TM starts, i.e. before any user-code is
> even accessible.
>
> My recommendation would be to either put the kafka dependencies in the
> /lib folder or try to relocate the kafka code in the reporter.
>
> On 11.07.2018 14:59, Gyula Fóra wrote:
> > Hi all,
> >
> > I have ran into the following problem and I want to double check
> > wether this is intended behaviour.
> >
> > I have a custom metrics reporter that pushes things to Kafka (so it
> > creates a KafkaProducer in the open method etc.etc.) for my streaming
> job.
> >
> > Naturally as my Flink job consumes from Kafka so it has the kafka
> > connector dependencies I set the Kafka dependencies to provided in my
> > metric reporter project and I put the built kafkaReporter.jar into the
> > Flink lib. However it seems that the metrics reporter is instantiated
> > without the user code classes since I get a NoClassdefFound error for
> > KafkaProducer even though my streaming job starts successfully
> > reading/writing kafka.
> >
> > Any ideas why this happens and how to solve it? I am slightly against
> > putting the kafka dependencies twice on the classpath as it has only
> > caused problems in the past...
> >
> > Gyula
>
>
>

Re: Custom metrics reporter classloading problem

Posted by Chesnay Schepler <ch...@apache.org>.
Reporters do not have access to libraries provided with user-jars.
They are instantiated when JM/TM starts, i.e. before any user-code is 
even accessible.

My recommendation would be to either put the kafka dependencies in the 
/lib folder or try to relocate the kafka code in the reporter.

On 11.07.2018 14:59, Gyula Fóra wrote:
> Hi all,
>
> I have ran into the following problem and I want to double check 
> wether this is intended behaviour.
>
> I have a custom metrics reporter that pushes things to Kafka (so it 
> creates a KafkaProducer in the open method etc.etc.) for my streaming job.
>
> Naturally as my Flink job consumes from Kafka so it has the kafka 
> connector dependencies I set the Kafka dependencies to provided in my 
> metric reporter project and I put the built kafkaReporter.jar into the 
> Flink lib. However it seems that the metrics reporter is instantiated 
> without the user code classes since I get a NoClassdefFound error for 
> KafkaProducer even though my streaming job starts successfully 
> reading/writing kafka.
>
> Any ideas why this happens and how to solve it? I am slightly against 
> putting the kafka dependencies twice on the classpath as it has only 
> caused problems in the past...
>
> Gyula