You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexey Trenikhun <ye...@msn.com> on 2022/04/20 22:00:50 UTC
Integration Test for Kafka Streaming job
Hello,
We have Flink job that read data from multiple Kafka topics, transforms data and write in output Kafka topics. We want write integration test for it. I've looked at KafkaTableITCase, we can do similar setup of Kafka topics, prepopulate data but since in our case it is endless stream, we need after some timeout (or condition on output topics) stop the job. Should we run 'env.execute("Job");` on background thread and then in main thread wait and then interrupt background thread? Or maybe there is dedicated method to stop/cancel job when needed. Are there better examples of Integration Tests for jobs with endless stream?
Thanks,
Alexey
Re: Integration Test for Kafka Streaming job
Posted by Alexey Trenikhun <ye...@msn.com>.
Thank you for information !
________________________________
From: Farouk <fa...@gmail.com>
Sent: Thursday, April 21, 2022 1:14:00 AM
To: Aeden Jameson <ae...@gmail.com>
Cc: Alexey Trenikhun <ye...@msn.com>; Flink User Mail List <us...@flink.apache.org>
Subject: Re: Integration Test for Kafka Streaming job
Hi
I would recommend to use kafka-junit5 from salesforce https://github.com/salesforce/kafka-junit
On top of that, you can use org.apache.flink.runtime.minicluster.TestingMiniCluster
Your stack should be complete.
Cheers
Le jeu. 21 avr. 2022 à 07:10, Aeden Jameson <ae...@gmail.com>> a écrit :
I've had success using Kafka for Junit,
https://github.com/mguenther/kafka-junit, for these kinds of tests.
On Wed, Apr 20, 2022 at 3:01 PM Alexey Trenikhun <ye...@msn.com>> wrote:
>
> Hello,
> We have Flink job that read data from multiple Kafka topics, transforms data and write in output Kafka topics. We want write integration test for it. I've looked at KafkaTableITCase, we can do similar setup of Kafka topics, prepopulate data but since in our case it is endless stream, we need after some timeout (or condition on output topics) stop the job. Should we run 'env.execute("Job");` on background thread and then in main thread wait and then interrupt background thread? Or maybe there is dedicated method to stop/cancel job when needed. Are there better examples of Integration Tests for jobs with endless stream?
>
> Thanks,
> Alexey
--
Cheers,
Aeden
GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson
Re: Integration Test for Kafka Streaming job
Posted by Farouk <fa...@gmail.com>.
Hi
I would recommend to use kafka-junit5 from salesforce
https://github.com/salesforce/kafka-junit
On top of that, you can
use org.apache.flink.runtime.minicluster.TestingMiniCluster
Your stack should be complete.
Cheers
Le jeu. 21 avr. 2022 à 07:10, Aeden Jameson <ae...@gmail.com> a
écrit :
> I've had success using Kafka for Junit,
> https://github.com/mguenther/kafka-junit, for these kinds of tests.
>
> On Wed, Apr 20, 2022 at 3:01 PM Alexey Trenikhun <ye...@msn.com> wrote:
> >
> > Hello,
> > We have Flink job that read data from multiple Kafka topics, transforms
> data and write in output Kafka topics. We want write integration test for
> it. I've looked at KafkaTableITCase, we can do similar setup of Kafka
> topics, prepopulate data but since in our case it is endless stream, we
> need after some timeout (or condition on output topics) stop the job.
> Should we run 'env.execute("Job");` on background thread and then in main
> thread wait and then interrupt background thread? Or maybe there is
> dedicated method to stop/cancel job when needed. Are there better examples
> of Integration Tests for jobs with endless stream?
> >
> > Thanks,
> > Alexey
>
>
>
> --
> Cheers,
> Aeden
>
> GitHub: https://github.com/aedenj
> Linked In: http://www.linkedin.com/in/aedenjameson
>
Re: Integration Test for Kafka Streaming job
Posted by Aeden Jameson <ae...@gmail.com>.
I've had success using Kafka for Junit,
https://github.com/mguenther/kafka-junit, for these kinds of tests.
On Wed, Apr 20, 2022 at 3:01 PM Alexey Trenikhun <ye...@msn.com> wrote:
>
> Hello,
> We have Flink job that read data from multiple Kafka topics, transforms data and write in output Kafka topics. We want write integration test for it. I've looked at KafkaTableITCase, we can do similar setup of Kafka topics, prepopulate data but since in our case it is endless stream, we need after some timeout (or condition on output topics) stop the job. Should we run 'env.execute("Job");` on background thread and then in main thread wait and then interrupt background thread? Or maybe there is dedicated method to stop/cancel job when needed. Are there better examples of Integration Tests for jobs with endless stream?
>
> Thanks,
> Alexey
--
Cheers,
Aeden
GitHub: https://github.com/aedenj
Linked In: http://www.linkedin.com/in/aedenjameson