You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Barak Ben-Nathan <ba...@xmcyber.com> on 2023/12/01 20:30:25 UTC

Fwd: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

hi,

I am trying to upgrade my app to Flink 1.18.

I have tests that run my stream from/to Embedded (in-memory) Kafka.
I.e. They create a flink cluster thus:

val flinkCluster = new MiniClusterWithClientResource(new
MiniClusterResourceConfiguration.Builder()
  .setNumberSlotsPerTaskManager(2)
  .setNumberTaskManagers(1)
  .build)

before {
  flinkCluster.before()
}

after {
  flinkCluster.after()
}

get an execution env thus:

val env = StreamExecutionEnvironment.getExecutionEnvironment

build the pipeline and execute it.

These tests fail due to:

org.apache.flink.runtime.client.JobInitializationException: Could not start
the JobMaster.

at
org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)

at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)

at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)

at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)

at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)

at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: java.util.concurrent.CompletionException:
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.flink.api.common.ExecutionConfig

at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)

at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)

at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)

... 3 more

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.flink.api.common.ExecutionConfig

at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)

at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)

at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)

... 3 more

Caused by: java.lang.ClassNotFoundException:
org.apache.flink.api.common.ExecutionConfig

at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)

at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)

at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)

at java.base/java.lang.Class.forName0(Native Method)

at java.base/java.lang.Class.forName(Class.java:398)

at
org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)

at
java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)

at
java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)

at
java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)

at
java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)

at
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)

at
java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)

at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)

at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)

at
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)

at
org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)

at
org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)

at
org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)

at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)

at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)

at
org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)

at
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
... 4 more

My flink dependencies are:

val flinkDependencies = Seq(
//  "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
//  "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
  "org.apache.flink" % "flink-connector-base" % "1.18.0",
  "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")

val testFlinkDependencies = Seq(
  // flink-test-utils brings log4j-slf4j-impl, which causes
java.lang.NoSuchMethodError when running tests
  // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
  "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
    ExclusionRule(organization = "org.apache.logging.log4j", name =
"log4j-slf4j-impl")
    ),
  "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
  "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test
classifier "tests"
)

 This used to work fine in Flink 1.13....

Does anyone have an idea what is causing this?

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Posted by Barak Ben-Nathan <ba...@xmcyber.com>.
Hi guys,

Forking in sbt solved the issue (Test / fork := true).


On Sun, Dec 3, 2023 at 7:48 AM Barak Ben-Nathan <ba...@xmcyber.com>
wrote:

> By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to
> no avail.
>
> On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan <ba...@xmcyber.com>
> wrote:
>
>> Thank's Jim,
>>
>> Unfortunately, this did not resolve the issue.
>>
>> I tried downgrading to 17.2 -- and everything works fine.
>> In version 18.0 -- I also added  flink-clients (in addition to
>> flink-core), but the problem persists....
>>
>>
>> On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user <
>> user@flink.apache.org> wrote:
>>
>>> Hi Barak,
>>>
>>> The missing class is in "flink-core", I think adding that dependency
>>> will provide it.
>>>
>>> The release notes for 1.14 note that Connectors no longer bundle
>>> "flink-core".  I imagine this is what has caused your issue.
>>>
>>> https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
>>>
>>> Cheers,
>>>
>>> Jim
>>>
>>> On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
>>> wrote:
>>>
>>>>
>>>> hi,
>>>>
>>>> I am trying to upgrade my app to Flink 1.18.
>>>>
>>>> I have tests that run my stream from/to Embedded (in-memory) Kafka.
>>>> I.e. They create a flink cluster thus:
>>>>
>>>> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>>>>   .setNumberSlotsPerTaskManager(2)
>>>>   .setNumberTaskManagers(1)
>>>>   .build)
>>>>
>>>> before {
>>>>   flinkCluster.before()
>>>> }
>>>>
>>>> after {
>>>>   flinkCluster.after()
>>>> }
>>>>
>>>> get an execution env thus:
>>>>
>>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>>
>>>> build the pipeline and execute it.
>>>>
>>>> These tests fail due to:
>>>>
>>>> org.apache.flink.runtime.client.JobInitializationException: Could not
>>>> start the JobMaster.
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>
>>>> Caused by: java.util.concurrent.CompletionException:
>>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>>> org.apache.flink.api.common.ExecutionConfig
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>>>>
>>>> ... 3 more
>>>>
>>>> Caused by: java.lang.RuntimeException:
>>>> java.lang.ClassNotFoundException:
>>>> org.apache.flink.api.common.ExecutionConfig
>>>>
>>>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>>>>
>>>> at
>>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>>>>
>>>> at
>>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>>>>
>>>> ... 3 more
>>>>
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> org.apache.flink.api.common.ExecutionConfig
>>>>
>>>> at
>>>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>>>>
>>>> at
>>>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>>>>
>>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>>>>
>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>>
>>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>>
>>>> at
>>>> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>>>>
>>>> at
>>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>>>>
>>>> at
>>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>>>>
>>>> at
>>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>>>>
>>>> at
>>>> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>>>>
>>>> at
>>>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>>>>
>>>> at
>>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>>>>
>>>> at
>>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
>>>> ... 4 more
>>>>
>>>> My flink dependencies are:
>>>>
>>>> val flinkDependencies = Seq(
>>>> //  "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>>>>   "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
>>>> //  "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>>>>   "org.apache.flink" % "flink-connector-base" % "1.18.0",
>>>>   "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>>>>
>>>> val testFlinkDependencies = Seq(
>>>>   // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>>>>   // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>>>>   "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>>>>     ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>>>>     ),
>>>>   "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>>>>   "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
>>>> )
>>>>
>>>>  This used to work fine in Flink 1.13....
>>>>
>>>> Does anyone have an idea what is causing this?
>>>>
>>>>
>>>>
>>>>
>>>>

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Posted by Barak Ben-Nathan <ba...@xmcyber.com>.
By the way, I also upgraded to flink-connector-kafka ver. 3.0.2-1.18, to no
avail.

On Sun, Dec 3, 2023 at 7:45 AM Barak Ben-Nathan <ba...@xmcyber.com>
wrote:

> Thank's Jim,
>
> Unfortunately, this did not resolve the issue.
>
> I tried downgrading to 17.2 -- and everything works fine.
> In version 18.0 -- I also added  flink-clients (in addition to
> flink-core), but the problem persists....
>
>
> On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user <us...@flink.apache.org>
> wrote:
>
>> Hi Barak,
>>
>> The missing class is in "flink-core", I think adding that dependency will
>> provide it.
>>
>> The release notes for 1.14 note that Connectors no longer bundle
>> "flink-core".  I imagine this is what has caused your issue.
>>
>> https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
>>
>> Cheers,
>>
>> Jim
>>
>> On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
>> wrote:
>>
>>>
>>> hi,
>>>
>>> I am trying to upgrade my app to Flink 1.18.
>>>
>>> I have tests that run my stream from/to Embedded (in-memory) Kafka.
>>> I.e. They create a flink cluster thus:
>>>
>>> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>>>   .setNumberSlotsPerTaskManager(2)
>>>   .setNumberTaskManagers(1)
>>>   .build)
>>>
>>> before {
>>>   flinkCluster.before()
>>> }
>>>
>>> after {
>>>   flinkCluster.after()
>>> }
>>>
>>> get an execution env thus:
>>>
>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>>
>>> build the pipeline and execute it.
>>>
>>> These tests fail due to:
>>>
>>> org.apache.flink.runtime.client.JobInitializationException: Could not
>>> start the JobMaster.
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>>>
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>
>>> Caused by: java.util.concurrent.CompletionException:
>>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.flink.api.common.ExecutionConfig
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>>>
>>> ... 3 more
>>>
>>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>>> org.apache.flink.api.common.ExecutionConfig
>>>
>>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>>>
>>> at
>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>>>
>>> at
>>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>>>
>>> ... 3 more
>>>
>>> Caused by: java.lang.ClassNotFoundException:
>>> org.apache.flink.api.common.ExecutionConfig
>>>
>>> at
>>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>>>
>>> at
>>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>>>
>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>>>
>>> at java.base/java.lang.Class.forName0(Native Method)
>>>
>>> at java.base/java.lang.Class.forName(Class.java:398)
>>>
>>> at
>>> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>>>
>>> at
>>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>>>
>>> at
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>>>
>>> at
>>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>>>
>>> at
>>> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>>>
>>> at
>>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>>>
>>> at
>>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>>>
>>> at
>>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
>>> ... 4 more
>>>
>>> My flink dependencies are:
>>>
>>> val flinkDependencies = Seq(
>>> //  "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>>>   "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
>>> //  "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>>>   "org.apache.flink" % "flink-connector-base" % "1.18.0",
>>>   "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>>>
>>> val testFlinkDependencies = Seq(
>>>   // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>>>   // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>>>   "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>>>     ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>>>     ),
>>>   "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>>>   "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
>>> )
>>>
>>>  This used to work fine in Flink 1.13....
>>>
>>> Does anyone have an idea what is causing this?
>>>
>>>
>>>
>>>
>>>

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Posted by Barak Ben-Nathan <ba...@xmcyber.com>.
Thank's Jim,

Unfortunately, this did not resolve the issue.

I tried downgrading to 17.2 -- and everything works fine.
In version 18.0 -- I also added  flink-clients (in addition to flink-core),
but the problem persists....


On Fri, Dec 1, 2023 at 10:44 PM Jim Hughes via user <us...@flink.apache.org>
wrote:

> Hi Barak,
>
> The missing class is in "flink-core", I think adding that dependency will
> provide it.
>
> The release notes for 1.14 note that Connectors no longer bundle
> "flink-core".  I imagine this is what has caused your issue.
>
> https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core
>
> Cheers,
>
> Jim
>
> On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
> wrote:
>
>>
>> hi,
>>
>> I am trying to upgrade my app to Flink 1.18.
>>
>> I have tests that run my stream from/to Embedded (in-memory) Kafka.
>> I.e. They create a flink cluster thus:
>>
>> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>>   .setNumberSlotsPerTaskManager(2)
>>   .setNumberTaskManagers(1)
>>   .build)
>>
>> before {
>>   flinkCluster.before()
>> }
>>
>> after {
>>   flinkCluster.after()
>> }
>>
>> get an execution env thus:
>>
>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>
>> build the pipeline and execute it.
>>
>> These tests fail due to:
>>
>> org.apache.flink.runtime.client.JobInitializationException: Could not
>> start the JobMaster.
>>
>> at
>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>>
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>
>> at java.base/java.lang.Thread.run(Thread.java:829)
>>
>> Caused by: java.util.concurrent.CompletionException:
>> java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.flink.api.common.ExecutionConfig
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>>
>> ... 3 more
>>
>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
>> org.apache.flink.api.common.ExecutionConfig
>>
>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>>
>> at
>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>>
>> at
>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>>
>> ... 3 more
>>
>> Caused by: java.lang.ClassNotFoundException:
>> org.apache.flink.api.common.ExecutionConfig
>>
>> at
>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>>
>> at
>> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>>
>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>>
>> at java.base/java.lang.Class.forName0(Native Method)
>>
>> at java.base/java.lang.Class.forName(Class.java:398)
>>
>> at
>> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>>
>> at
>> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>>
>> at
>> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>>
>> at
>> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>>
>> at
>> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>>
>> at
>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>>
>> at
>> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>>
>> at
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>>
>> at
>> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>>
>> at
>> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>>
>> at
>> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>>
>> at
>> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>>
>> at
>> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>>
>> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>>
>> at
>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>>
>> at
>> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>>
>> at
>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
>> ... 4 more
>>
>> My flink dependencies are:
>>
>> val flinkDependencies = Seq(
>> //  "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>>   "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
>> //  "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>>   "org.apache.flink" % "flink-connector-base" % "1.18.0",
>>   "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>>
>> val testFlinkDependencies = Seq(
>>   // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>>   // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>>   "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>>     ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>>     ),
>>   "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>>   "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
>> )
>>
>>  This used to work fine in Flink 1.13....
>>
>> Does anyone have an idea what is causing this?
>>
>>
>>
>>
>>

Re: Getting java.lang.ClassNotFoundException in Tests (Flink 1.18.0)

Posted by Jim Hughes via user <us...@flink.apache.org>.
Hi Barak,

The missing class is in "flink-core", I think adding that dependency will
provide it.

The release notes for 1.14 note that Connectors no longer bundle
"flink-core".  I imagine this is what has caused your issue.
https://nightlies.apache.org/flink/flink-docs-release-1.18/release-notes/flink-1.14/#connector-base-exposes-dependency-to-flink-core

Cheers,

Jim

On Fri, Dec 1, 2023 at 3:30 PM Barak Ben-Nathan <ba...@xmcyber.com>
wrote:

>
> hi,
>
> I am trying to upgrade my app to Flink 1.18.
>
> I have tests that run my stream from/to Embedded (in-memory) Kafka.
> I.e. They create a flink cluster thus:
>
> val flinkCluster = new MiniClusterWithClientResource(new MiniClusterResourceConfiguration.Builder()
>   .setNumberSlotsPerTaskManager(2)
>   .setNumberTaskManagers(1)
>   .build)
>
> before {
>   flinkCluster.before()
> }
>
> after {
>   flinkCluster.after()
> }
>
> get an execution env thus:
>
> val env = StreamExecutionEnvironment.getExecutionEnvironment
>
> build the pipeline and execute it.
>
> These tests fail due to:
>
> org.apache.flink.runtime.client.JobInitializationException: Could not
> start the JobMaster.
>
> at
> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
>
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
>
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
>
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
>
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1705)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:829)
>
> Caused by: java.util.concurrent.CompletionException:
> java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.flink.api.common.ExecutionConfig
>
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
>
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
>
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
>
> ... 3 more
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> org.apache.flink.api.common.ExecutionConfig
>
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
>
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
>
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
>
> ... 3 more
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.api.common.ExecutionConfig
>
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
>
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
>
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
>
> at java.base/java.lang.Class.forName0(Native Method)
>
> at java.base/java.lang.Class.forName(Class.java:398)
>
> at
> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.resolveClass(InstantiationUtil.java:78)
>
> at
> java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
>
> at
> java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
>
> at
> java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
>
> at
> java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
>
> at
> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
>
> at
> java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
>
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
>
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
>
> at
> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67)
>
> at
> org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:101)
>
> at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:122)
>
> at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:379)
>
> at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:356)
>
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:128)
>
> at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:100)
>
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> ... 4 more
>
> My flink dependencies are:
>
> val flinkDependencies = Seq(
> //  "org.apache.flink" %% "flink-scala" % flinkVersion % Provided,
>   "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % Provided,
> //  "org.apache.flink" % "flink-streaming-java" % flinkVersion % Provided,
>   "org.apache.flink" % "flink-connector-base" % "1.18.0",
>   "org.apache.flink" % "flink-connector-kafka" % "3.0.1-1.18")
>
> val testFlinkDependencies = Seq(
>   // flink-test-utils brings log4j-slf4j-impl, which causes java.lang.NoSuchMethodError when running tests
>   // https://stackoverflow.com/questions/75710149/java-lang-nosuchmethoderror-void-org-apache-logging-slf4j-log4jloggerfactory
>   "org.apache.flink" % "flink-test-utils" % flinkVersion % Test excludeAll (
>     ExclusionRule(organization = "org.apache.logging.log4j", name = "log4j-slf4j-impl")
>     ),
>   "org.apache.flink" % "flink-runtime" % flinkVersion % Test,
>   "org.apache.flink" % "flink-streaming-java" % flinkVersion % Test classifier "tests"
> )
>
>  This used to work fine in Flink 1.13....
>
> Does anyone have an idea what is causing this?
>
>
>
>
>