You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Barak Ben-Nathan <ba...@xmcyber.com> on 2023/12/01 20:30:25 UTC
Fwd: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)
hi,
I am trying to upgrade my app to Flink 1.18.
I have tests that run my stream from/to Embedded (in-memory) Kafka.
I.e. They create a flink cluster thus:
val flinkCluster = new MiniClusterWithClientResource(new
MiniClusterResourceConfiguration.Builder()
.setNumberSlotsPerTaskManager(2)
.setNumberTaskManagers(1)
.build)
before {
flinkCluster.before()
}
after {
flinkCluster.after()
}
get an execution env thus:
val env = StreamExecutionEnvironment.getExecutionEnvironment
build the pipeline and execute it.
These tests fail due to:
org.apache.flink.runtime.client.JobInitializationException: Could not start
the JobMaster.
at
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.CompletionException:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.flink.api.common.ExecutionConfig
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
... 3 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.flink.api.common.ExecutionConfig
at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
... 3 more
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.api.common.ExecutionConfig
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
at
java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
at
java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
at
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
at
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
at
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
at
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
at
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
at
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
... 4 more
My flink dependencies are:
val flinkDependencies = Seq(
// "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
// "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
"org.apache.flink" % "flink-connector-base" % "1.18.0",
"org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
val testFlinkDependencies = Seq(
// flink-test-utils brings log4j-slf4j-impl, which causes
java.lang.NoSuchMethodError when running tests
// https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
"org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
ExclusionRule(organization = "org.apache.logging.log4j", name =
"log4j-slf4j-impl")
),
"org.apache.flink" % "flink-runtime" % flinkVersion % Test,
"org.apache.flink" % "flink-streaming-java" % flinkVersion % Test
classifier "tests"
)
This used to work fine in Flink 1.13....
Does anyone have an idea what is causing this?
Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)
Posted by Barak Ben-Nathan <ba...@xmcyber.com>.
Hi guys,
Forking in sbt solved the issue (Test / fork := true).
On Sun, Dec 3, 2023 at 7:48 AM Barak Ben-Nathan <ba...@xmcyber.com>
wrote:
> By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to
> no avail.
>
> On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan <ba...@xmcyber.com>
> wrote:
>
>> Thank's Jim,
>>
>> Unfortunately, this did not resolve the issue.
>>
>> I tried downgrading to 17.2 -- and everything works fine.
>> In version 18.0 -- I also added flink-clients (in addition to
>> flink-core), but the problem persists....
>>
>>
>> On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user <
>> user@flink.apache.org> wrote:
>>
>>> Hi Barak,
>>>
>>> The missing class is in "flink-core", I think adding that dependency
>>> will provide it.
>>>
>>> The release notes for 1.14 note that Connectors no longer bundle
>>> "flink-core". I imagine this is what has caused your issue.
>>>
>>> https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
>>>
>>> Cheers,
>>>
>>> Jim
>>>
>>> On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
>>> wrote:
>>>
>>>>
>>>> hi,
>>>>
>>>> I am trying to upgrade my app to Flink 1.18.
>>>>
>>>> I have tests that run my stream from/to Embedded (in-memory) Kafka.
>>>> I.e. They create a flink cluster thus:
>>>>
>>>> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>>>> .setNumberSlotsPerTaskManager(2)
>>>> .setNumberTaskManagers(1)
>>>> .build)
>>>>
>>>> before {
>>>> flinkCluster.before()
>>>> }
>>>>
>>>> after {
>>>> flinkCluster.after()
>>>> }
>>>>
>>>> get an execution env thus:
>>>>
>>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>>
>>>> build the pipeline and execute it.
>>>>
>>>> These tests fail due to:
>>>>
>>>> org.apache.flink.runtime.client.JobInitializationException: Could not
>>>> start the JobMaster.
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>
>>>> Caused by: java.util.concurrent.CompletionException:
>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>>> org.apache.flink.api.common.ExecutionConfig
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>>>>
>>>> ... 3 more
>>>>
>>>> Caused by: java.lang.RuntimeException:
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.flink.api.common.ExecutionConfig
>>>>
>>>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>>>>
>>>> at
>>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>>>>
>>>> ... 3 more
>>>>
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.flink.api.common.ExecutionConfig
>>>>
>>>> at
>>>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>>>>
>>>> at
>>>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>>>>
>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>>>>
>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>
>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>
>>>> at
>>>> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>>>>
>>>> at
>>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>>>>
>>>> at
>>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>>>>
>>>> at
>>>> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>>>>
>>>> at
>>>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>>>>
>>>> at
>>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
>>>> ... 4 more
>>>>
>>>> My flink dependencies are:
>>>>
>>>> val flinkDependencies = Seq(
>>>> // "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>>>> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
>>>> // "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>>>> "org.apache.flink" % "flink-connector-base" % "1.18.0",
>>>> "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>>>>
>>>> val testFlinkDependencies = Seq(
>>>> // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>>>> // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>>>> "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>>>> ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>>>> ),
>>>> "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>>>> "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
>>>> )
>>>>
>>>> This used to work fine in Flink 1.13....
>>>>
>>>> Does anyone have an idea what is causing this?
>>>>
>>>>
>>>>
>>>>
>>>>
Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)
Posted by Barak Ben-Nathan <ba...@xmcyber.com>.
By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to no
avail.
On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan <ba...@xmcyber.com>
wrote:
> Thank's Jim,
>
> Unfortunately, this did not resolve the issue.
>
> I tried downgrading to 17.2 -- and everything works fine.
> In version 18.0 -- I also added flink-clients (in addition to
> flink-core), but the problem persists....
>
>
> On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user <us...@flink.apache.org>
> wrote:
>
>> Hi Barak,
>>
>> The missing class is in "flink-core", I think adding that dependency will
>> provide it.
>>
>> The release notes for 1.14 note that Connectors no longer bundle
>> "flink-core". I imagine this is what has caused your issue.
>>
>> https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
>>
>> Cheers,
>>
>> Jim
>>
>> On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
>> wrote:
>>
>>>
>>> hi,
>>>
>>> I am trying to upgrade my app to Flink 1.18.
>>>
>>> I have tests that run my stream from/to Embedded (in-memory) Kafka.
>>> I.e. They create a flink cluster thus:
>>>
>>> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>>> .setNumberSlotsPerTaskManager(2)
>>> .setNumberTaskManagers(1)
>>> .build)
>>>
>>> before {
>>> flinkCluster.before()
>>> }
>>>
>>> after {
>>> flinkCluster.after()
>>> }
>>>
>>> get an execution env thus:
>>>
>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>
>>> build the pipeline and execute it.
>>>
>>> These tests fail due to:
>>>
>>> org.apache.flink.runtime.client.JobInitializationException: Could not
>>> start the JobMaster.
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>>>
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>
>>> Caused by: java.util.concurrent.CompletionException:
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.flink.api.common.ExecutionConfig
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>>>
>>> ... 3 more
>>>
>>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.flink.api.common.ExecutionConfig
>>>
>>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>>>
>>> at
>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>>>
>>> ... 3 more
>>>
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.flink.api.common.ExecutionConfig
>>>
>>> at
>>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>>>
>>> at
>>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>>>
>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>>>
>>> at java.base/java.lang.Class.forName0(Native Method)
>>>
>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>
>>> at
>>> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>>>
>>> at
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>>>
>>> at
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>>>
>>> at
>>> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>>>
>>> at
>>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>>>
>>> at
>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
>>> ... 4 more
>>>
>>> My flink dependencies are:
>>>
>>> val flinkDependencies = Seq(
>>> // "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>>> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
>>> // "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>>> "org.apache.flink" % "flink-connector-base" % "1.18.0",
>>> "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>>>
>>> val testFlinkDependencies = Seq(
>>> // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>>> // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>>> "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>>> ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>>> ),
>>> "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>>> "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
>>> )
>>>
>>> This used to work fine in Flink 1.13....
>>>
>>> Does anyone have an idea what is causing this?
>>>
>>>
>>>
>>>
>>>
Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)
Posted by Barak Ben-Nathan <ba...@xmcyber.com>.
Thank's Jim,
Unfortunately, this did not resolve the issue.
I tried downgrading to 17.2 -- and everything works fine.
In version 18.0 -- I also added flink-clients (in addition to flink-core),
but the problem persists....
On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user <us...@flink.apache.org>
wrote:
> Hi Barak,
>
> The missing class is in "flink-core", I think adding that dependency will
> provide it.
>
> The release notes for 1.14 note that Connectors no longer bundle
> "flink-core". I imagine this is what has caused your issue.
>
> https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
>
> Cheers,
>
> Jim
>
> On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
> wrote:
>
>>
>> hi,
>>
>> I am trying to upgrade my app to Flink 1.18.
>>
>> I have tests that run my stream from/to Embedded (in-memory) Kafka.
>> I.e. They create a flink cluster thus:
>>
>> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>> .setNumberSlotsPerTaskManager(2)
>> .setNumberTaskManagers(1)
>> .build)
>>
>> before {
>> flinkCluster.before()
>> }
>>
>> after {
>> flinkCluster.after()
>> }
>>
>> get an execution env thus:
>>
>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>
>> build the pipeline and execute it.
>>
>> These tests fail due to:
>>
>> org.apache.flink.runtime.client.JobInitializationException: Could not
>> start the JobMaster.
>>
>> at
>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>>
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>
>> at java.base/java.lang.Thread.run(Thread.java:829)
>>
>> Caused by: java.util.concurrent.CompletionException:
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.flink.api.common.ExecutionConfig
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>>
>> ... 3 more
>>
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.flink.api.common.ExecutionConfig
>>
>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>>
>> at
>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>>
>> ... 3 more
>>
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.flink.api.common.ExecutionConfig
>>
>> at
>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>>
>> at
>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>>
>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>>
>> at java.base/java.lang.Class.forName0(Native Method)
>>
>> at java.base/java.lang.Class.forName(Class.java:398)
>>
>> at
>> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>>
>> at
>> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>>
>> at
>> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>>
>> at
>> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>>
>> at
>> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>>
>> at
>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>>
>> at
>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>>
>> at
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>>
>> at
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>>
>> at
>> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>>
>> at
>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>>
>> at
>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>>
>> at
>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>>
>> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>>
>> at
>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>>
>> at
>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>>
>> at
>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
>> ... 4 more
>>
>> My flink dependencies are:
>>
>> val flinkDependencies = Seq(
>> // "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
>> // "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>> "org.apache.flink" % "flink-connector-base" % "1.18.0",
>> "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>>
>> val testFlinkDependencies = Seq(
>> // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>> // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>> "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>> ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>> ),
>> "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>> "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
>> )
>>
>> This used to work fine in Flink 1.13....
>>
>> Does anyone have an idea what is causing this?
>>
>>
>>
>>
>>
Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)
Posted by Jim Hughes via user <us...@flink.apache.org>.
Hi Barak,
The missing class is in "flink-core", I think adding that dependency will
provide it.
The release notes for 1.14 note that Connectors no longer bundle
"flink-core". I imagine this is what has caused your issue.
https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
Cheers,
Jim
On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
wrote:
>
> hi,
>
> I am trying to upgrade my app to Flink 1.18.
>
> I have tests that run my stream from/to Embedded (in-memory) Kafka.
> I.e. They create a flink cluster thus:
>
> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
> .setNumberSlotsPerTaskManager(2)
> .setNumberTaskManagers(1)
> .build)
>
> before {
> flinkCluster.before()
> }
>
> after {
> flinkCluster.after()
> }
>
> get an execution env thus:
>
> val env = StreamExecutionEnvironment.getExecutionEnvironment
>
> build the pipeline and execute it.
>
> These tests fail due to:
>
> org.apache.flink.runtime.client.JobInitializationException: Could not
> start the JobMaster.
>
> at
> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> Caused by: java.util.concurrent.CompletionException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.flink.api.common.ExecutionConfig
>
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>
> ... 3 more
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.flink.api.common.ExecutionConfig
>
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>
> ... 3 more
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.api.common.ExecutionConfig
>
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>
> at java.base/java.lang.Class.forName0(Native Method)
>
> at java.base/java.lang.Class.forName(Class.java:398)
>
> at
> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>
> at
> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>
> at
> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>
> at
> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>
> at
> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>
> at
> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>
> at
> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>
> at
> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>
> at
> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>
> at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>
> at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>
> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> ... 4 more
>
> My flink dependencies are:
>
> val flinkDependencies = Seq(
> // "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
> "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
> // "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
> "org.apache.flink" % "flink-connector-base" % "1.18.0",
> "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>
> val testFlinkDependencies = Seq(
> // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
> // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
> "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
> ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
> ),
> "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
> "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
> )
>
> This used to work fine in Flink 1.13....
>
> Does anyone have an idea what is causing this?
>
>
>
>
>