You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Farouk <fa...@gmail.com> on 2021/07/13 11:21:06 UTC
Running two versions of Flink with testcontainers
Hi
For e2e testing, we run tests with testcontainers. We have several jobs and
we want to upgrade them one by one
Do you know if it is possible in Docker to run one JM + one TM for version
1 and version 2 at the same time?
It looks like either the taskmanager registration is failing for the second
cluster, either there is a classcastexception
ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at
ResourceManager failed due to an error
java.util.concurrent.CompletionException: java.lang.ClassCastException:
class org.apache.flink.util.SerializedValue cannot be cast to class
org.apache.flink.runtime.registration.RegistrationResponse
(org.apache.flink.util.SerializedValue and
org.apache.flink.runtime.registration.RegistrationResponse are in unnamed
module of loader 'app')
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
Source)
at
java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown
Source)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown
Source)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.ClassCastException: class
org.apache.flink.util.SerializedValue cannot be cast to class
org.apache.flink.runtime.registration.RegistrationResponse
(org.apache.flink.util.SerializedValue and
org.apache.flink.runtime.registration.RegistrationResponse are in unnamed
module of loader 'app')
... 8 common frames omitted
Thanks
Farouk
Re: Running two versions of Flink with testcontainers
Posted by Austin Cawley-Edwards <au...@gmail.com>.
Great, glad it worked out for you!
On Tue, Jul 13, 2021 at 10:32 AM Farouk <fa...@gmail.com> wrote:
> Thanks
>
> Finally I tried by running docker commands (thanks for the documentation)
> and it works fine.
>
> Thanks
> Farouk
>
> Le mar. 13 juil. 2021 à 15:48, Austin Cawley-Edwards <
> austin.cawley@gmail.com> a écrit :
>
>> You might also be able to put them in separate networks[1] to get around
>> changing all the ports and still ensuring that they don't see eachother.
>>
>> [1]:
>> https://www.testcontainers.org/features/networking#advanced-networking
>>
>> On Tue, Jul 13, 2021 at 9:07 AM Chesnay Schepler <ch...@apache.org>
>> wrote:
>>
>>> It is possible but you need to make sure that all ports a configured
>>> such that the 2 clusters don't see each other.
>>>
>>> On 13/07/2021 13:21, Farouk wrote:
>>> > Hi
>>> >
>>> > For e2e testing, we run tests with testcontainers. We have several
>>> > jobs and we want to upgrade them one by one
>>> >
>>> > Do you know if it is possible in Docker to run one JM + one TM for
>>> > version 1 and version 2 at the same time?
>>> >
>>> > It looks like either the taskmanager registration is failing for the
>>> > second cluster, either there is a classcastexception
>>> >
>>> > ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor -
>>> > Registration at ResourceManager failed due to an error
>>> > java.util.concurrent.CompletionException:
>>> > java.lang.ClassCastException: class
>>> > org.apache.flink.util.SerializedValue cannot be cast to class
>>> > org.apache.flink.runtime.registration.RegistrationResponse
>>> > (org.apache.flink.util.SerializedValue and
>>> > org.apache.flink.runtime.registration.RegistrationResponse are in
>>> > unnamed module of loader 'app')
>>> > at
>>> >
>>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
>>> > Source)
>>> > at
>>> >
>>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
>>> > Source)
>>> > at
>>> >
>>> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown
>>> > Source)
>>> > at
>>> >
>>> java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown
>>> > Source)
>>> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>>> > at
>>> >
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
>>> > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>> > at
>>> >
>>> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>> > at
>>> akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>> > at
>>> >
>>> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> > Caused by: java.lang.ClassCastException: class
>>> > org.apache.flink.util.SerializedValue cannot be cast to class
>>> > org.apache.flink.runtime.registration.RegistrationResponse
>>> > (org.apache.flink.util.SerializedValue and
>>> > org.apache.flink.runtime.registration.RegistrationResponse are in
>>> > unnamed module of loader 'app')
>>> > ... 8 common frames omitted
>>> >
>>> >
>>> > Thanks
>>> > Farouk
>>>
>>>
>>>
Re: Running two versions of Flink with testcontainers
Posted by Austin Cawley-Edwards <au...@gmail.com>.
You might also be able to put them in separate networks[1] to get around
changing all the ports and still ensuring that they don't see eachother.
[1]: https://www.testcontainers.org/features/networking#advanced-networking
On Tue, Jul 13, 2021 at 9:07 AM Chesnay Schepler <ch...@apache.org> wrote:
> It is possible but you need to make sure that all ports a configured
> such that the 2 clusters don't see each other.
>
> On 13/07/2021 13:21, Farouk wrote:
> > Hi
> >
> > For e2e testing, we run tests with testcontainers. We have several
> > jobs and we want to upgrade them one by one
> >
> > Do you know if it is possible in Docker to run one JM + one TM for
> > version 1 and version 2 at the same time?
> >
> > It looks like either the taskmanager registration is failing for the
> > second cluster, either there is a classcastexception
> >
> > ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor -
> > Registration at ResourceManager failed due to an error
> > java.util.concurrent.CompletionException:
> > java.lang.ClassCastException: class
> > org.apache.flink.util.SerializedValue cannot be cast to class
> > org.apache.flink.runtime.registration.RegistrationResponse
> > (org.apache.flink.util.SerializedValue and
> > org.apache.flink.runtime.registration.RegistrationResponse are in
> > unnamed module of loader 'app')
> > at
> > java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
> > Source)
> > at
> >
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
> > Source)
> > at
> >
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown
> > Source)
> > at
> > java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown
> > Source)
> > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> > at
> >
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
> > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > at
> >
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > at
> >
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > Caused by: java.lang.ClassCastException: class
> > org.apache.flink.util.SerializedValue cannot be cast to class
> > org.apache.flink.runtime.registration.RegistrationResponse
> > (org.apache.flink.util.SerializedValue and
> > org.apache.flink.runtime.registration.RegistrationResponse are in
> > unnamed module of loader 'app')
> > ... 8 common frames omitted
> >
> >
> > Thanks
> > Farouk
>
>
>
Re: Running two versions of Flink with testcontainers
Posted by Chesnay Schepler <ch...@apache.org>.
It is possible but you need to make sure that all ports a configured
such that the 2 clusters don't see each other.
On 13/07/2021 13:21, Farouk wrote:
> Hi
>
> For e2e testing, we run tests with testcontainers. We have several
> jobs and we want to upgrade them one by one
>
> Do you know if it is possible in Docker to run one JM + one TM for
> version 1 and version 2 at the same time?
>
> It looks like either the taskmanager registration is failing for the
> second cluster, either there is a classcastexception
>
> ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor -
> Registration at ResourceManager failed due to an error
> java.util.concurrent.CompletionException:
> java.lang.ClassCastException: class
> org.apache.flink.util.SerializedValue cannot be cast to class
> org.apache.flink.runtime.registration.RegistrationResponse
> (org.apache.flink.util.SerializedValue and
> org.apache.flink.runtime.registration.RegistrationResponse are in
> unnamed module of loader 'app')
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown
> Source)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.ClassCastException: class
> org.apache.flink.util.SerializedValue cannot be cast to class
> org.apache.flink.runtime.registration.RegistrationResponse
> (org.apache.flink.util.SerializedValue and
> org.apache.flink.runtime.registration.RegistrationResponse are in
> unnamed module of loader 'app')
> ... 8 common frames omitted
>
>
> Thanks
> Farouk