You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Javier Vegas <jv...@strava.com> on 2022/01/26 03:13:07 UTC
IllegalArgumentException: URI is not hierarchical error when initializating jobmanager in cluster
I am porting a Scala service to Flink in order to make it more scalable via
running it in a cluster. All my Scala services extends a base Service class
that extends TwitterServer (
https://github.com/twitter/twitter-server/blob/develop/server/src/main/scala/com/twitter/server/TwitterServer.scala)
and that base class contains a lot of logic about resource initialization,
logging, stats and error handling, monitoring, etc that I want to keep
using in my class. I ported my logic to Flink sources and sinks, and
everything worked fine when I ran my class in single JVM mode either from
sbt or my IDE, Flink's jobmanager and taskmanagers start and run my app.
But when I try to run my application in cluster mode, when launching my
class with "./bin/standalone-job.sh start --job-classname" the jobmanager
runs into a "IllegalArgumentException: URI is not hierarchical" error on
initialization, apparently because TwitterServer is trying to load
something from the class path (see attached full log).
Is there anything I can do to run a class that extends TwitterServer in a
Flink cluster? I have tried making my class not extend it and it worked
fine, but I really want to keep using all the common infraestructure logic
that I have in my base class that extends TwitterServer.
Thanks!
Re: IllegalArgumentException: URI is not hierarchical error when initializating jobmanager in cluster
Posted by Javier Vegas <jv...@strava.com>.
Thanks, Robert!
I tried the classloader.resolve.order: parent-first option but ran into
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder" errors
(because I use logback so I followed
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/logging/#configuring-logback
and removed log4j-slf4j-impl from the classpath. But putting all my classes
in lib/ instead of usrlib/ fixed that problem, and everything now runs
fine. Thanks!
El vie, 28 ene 2022 a las 6:11, Robert Metzger (<me...@gmail.com>)
escribió:
> Hi Javier,
>
> I suspect that TwitterServer is using some classloading / dependency
> injection / service loading "magic" that is causing this.
> I would try to find out, either by attaching a remote debugger (should be
> possible when executing in cluster mode locally) or by adding log
> statements in the code, what the URI it's trying to load looks like.
>
> On the cluster, Flink is using separate classloaders for the base flink
> system, and the user code (as opposed to executing in the IDE, where
> everything is loaded from the same loader). Check out this page and try out
> the config arguments:
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/
>
>
>
> On Wed, Jan 26, 2022 at 4:13 AM Javier Vegas <jv...@strava.com> wrote:
>
>> I am porting a Scala service to Flink in order to make it more scalable
>> via running it in a cluster. All my Scala services extends a base Service
>> class that extends TwitterServer (
>> https://github.com/twitter/twitter-server/blob/develop/server/src/main/scala/com/twitter/server/TwitterServer.scala)
>> and that base class contains a lot of logic about resource initialization,
>> logging, stats and error handling, monitoring, etc that I want to keep
>> using in my class. I ported my logic to Flink sources and sinks, and
>> everything worked fine when I ran my class in single JVM mode either from
>> sbt or my IDE, Flink's jobmanager and taskmanagers start and run my app.
>> But when I try to run my application in cluster mode, when launching my
>> class with "./bin/standalone-job.sh start --job-classname" the
>> jobmanager runs into a "IllegalArgumentException: URI is not hierarchical"
>> error on initialization, apparently because TwitterServer is trying to load
>> something from the class path (see attached full log).
>>
>> Is there anything I can do to run a class that extends TwitterServer in a
>> Flink cluster? I have tried making my class not extend it and it worked
>> fine, but I really want to keep using all the common infraestructure logic
>> that I have in my base class that extends TwitterServer.
>>
>> Thanks!
>>
>
Re: IllegalArgumentException: URI is not hierarchical error when initializating jobmanager in cluster
Posted by Robert Metzger <me...@gmail.com>.
Hi Javier,
I suspect that TwitterServer is using some classloading / dependency
injection / service loading "magic" that is causing this.
I would try to find out, either by attaching a remote debugger (should be
possible when executing in cluster mode locally) or by adding log
statements in the code, what the URI it's trying to load looks like.
On the cluster, Flink is using separate classloaders for the base flink
system, and the user code (as opposed to executing in the IDE, where
everything is loaded from the same loader). Check out this page and try out
the config arguments:
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/
On Wed, Jan 26, 2022 at 4:13 AM Javier Vegas <jv...@strava.com> wrote:
> I am porting a Scala service to Flink in order to make it more scalable
> via running it in a cluster. All my Scala services extends a base Service
> class that extends TwitterServer (
> https://github.com/twitter/twitter-server/blob/develop/server/src/main/scala/com/twitter/server/TwitterServer.scala)
> and that base class contains a lot of logic about resource initialization,
> logging, stats and error handling, monitoring, etc that I want to keep
> using in my class. I ported my logic to Flink sources and sinks, and
> everything worked fine when I ran my class in single JVM mode either from
> sbt or my IDE, Flink's jobmanager and taskmanagers start and run my app.
> But when I try to run my application in cluster mode, when launching my
> class with "./bin/standalone-job.sh start --job-classname" the jobmanager
> runs into a "IllegalArgumentException: URI is not hierarchical" error on
> initialization, apparently because TwitterServer is trying to load
> something from the class path (see attached full log).
>
> Is there anything I can do to run a class that extends TwitterServer in a
> Flink cluster? I have tried making my class not extend it and it worked
> fine, but I really want to keep using all the common infraestructure logic
> that I have in my base class that extends TwitterServer.
>
> Thanks!
>