You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2018/07/11 12:59:16 UTC
Custom metrics reporter classloading problem
Hi all,
I have ran into the following problem and I want to double check wether
this is intended behaviour.
I have a custom metrics reporter that pushes things to Kafka (so it creates
a KafkaProducer in the open method etc.etc.) for my streaming job.
Naturally as my Flink job consumes from Kafka so it has the kafka connector
dependencies I set the Kafka dependencies to provided in my metric reporter
project and I put the built kafkaReporter.jar into the Flink lib. However
it seems that the metrics reporter is instantiated without the user code
classes since I get a NoClassdefFound error for KafkaProducer even though
my streaming job starts successfully reading/writing kafka.
Any ideas why this happens and how to solve it? I am slightly against
putting the kafka dependencies twice on the classpath as it has only caused
problems in the past...
Gyula
Re: Custom metrics reporter classloading problem
Posted by Gyula Fóra <gy...@gmail.com>.
Thanks for the explanation, that makes sense.
For some reason I thought that in Yarn all stuff goes into the classpath.
Gy
Chesnay Schepler <ch...@apache.org> ezt írta (időpont: 2018. júl. 11.,
Sze, 15:16):
> Reporters do not have access to libraries provided with user-jars.
> They are instantiated when JM/TM starts, i.e. before any user-code is
> even accessible.
>
> My recommendation would be to either put the kafka dependencies in the
> /lib folder or try to relocate the kafka code in the reporter.
>
> On 11.07.2018 14:59, Gyula Fóra wrote:
> > Hi all,
> >
> > I have ran into the following problem and I want to double check
> > wether this is intended behaviour.
> >
> > I have a custom metrics reporter that pushes things to Kafka (so it
> > creates a KafkaProducer in the open method etc.etc.) for my streaming
> job.
> >
> > Naturally as my Flink job consumes from Kafka so it has the kafka
> > connector dependencies I set the Kafka dependencies to provided in my
> > metric reporter project and I put the built kafkaReporter.jar into the
> > Flink lib. However it seems that the metrics reporter is instantiated
> > without the user code classes since I get a NoClassdefFound error for
> > KafkaProducer even though my streaming job starts successfully
> > reading/writing kafka.
> >
> > Any ideas why this happens and how to solve it? I am slightly against
> > putting the kafka dependencies twice on the classpath as it has only
> > caused problems in the past...
> >
> > Gyula
>
>
>
Re: Custom metrics reporter classloading problem
Posted by Chesnay Schepler <ch...@apache.org>.
Reporters do not have access to libraries provided with user-jars.
They are instantiated when JM/TM starts, i.e. before any user-code is
even accessible.
My recommendation would be to either put the kafka dependencies in the
/lib folder or try to relocate the kafka code in the reporter.
On 11.07.2018 14:59, Gyula Fóra wrote:
> Hi all,
>
> I have ran into the following problem and I want to double check
> wether this is intended behaviour.
>
> I have a custom metrics reporter that pushes things to Kafka (so it
> creates a KafkaProducer in the open method etc.etc.) for my streaming job.
>
> Naturally as my Flink job consumes from Kafka so it has the kafka
> connector dependencies I set the Kafka dependencies to provided in my
> metric reporter project and I put the built kafkaReporter.jar into the
> Flink lib. However it seems that the metrics reporter is instantiated
> without the user code classes since I get a NoClassdefFound error for
> KafkaProducer even though my streaming job starts successfully
> reading/writing kafka.
>
> Any ideas why this happens and how to solve it? I am slightly against
> putting the kafka dependencies twice on the classpath as it has only
> caused problems in the past...
>
> Gyula