You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Laurent Exsteens <la...@euranova.eu> on 2020/03/02 11:26:50 UTC
Correct way to e2e test a Flink application?
Hello,
I would like to test a Flink application, including any problem that would
happen when deployed on a distributed cluster.
The way we do this currently is to launch a Flink cluster in Docker and run
the job on it. This setup seems heavy and might not be necessary.
Is there a way to simulate a Flink cluster as if it would be distributed
(including serialisation and static classes behavior on such a cluster)
using, for example, a Flink Mini cluster? If yes, is there any resources
available on how to setup a correct test environment?
Thanks in advance for your help.
Regards,
Laurent.
--
♻ Be green, keep it on the screen
Re: Correct way to e2e test a Flink application?
Posted by Laurent Exsteens <la...@euranova.eu>.
Hello Robert,
thanks a lot, I'll sure have a look at it!
Regards,
Laurent.
On Thu, 2 Apr 2020 at 10:54, Robert Metzger <rm...@apache.org> wrote:
> Hey Laurent,
> Flink developed an internal framework for executing end to end tests from
> Java. Here's an example of such a test:
> https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java#L98
> This should download and set up a real, distributed (on your machine, but
> in different processes) Flink cluster.
>
> Best,
> Robert
>
> On Tue, Mar 31, 2020 at 10:48 AM Laurent Exsteens <
> laurent.exsteens@euranova.eu> wrote:
>
>> Hi Tzu-Li,
>>
>> thanks a lot for your answer. I will try this!
>>
>> However, I was looking for something that does fully simulate a Flink
>> cluster, including job-manager to task manager serialization issues and
>> full isolation of the task managers (I guess in the MiniClusterResource, we
>> are still on the same JVM, so static classes will still be available to all
>> operators, which is different than on a real cluster).
>>
>> Cheers,
>>
>> Laurent.
>>
>> On Mon, 2 Mar 2020 at 13:24, Tzu-Li (Gordon) Tai <tz...@apache.org>
>> wrote:
>>
>>> Hi Laurent,
>>>
>>> You can take a look at Flink's MiniClusterResource JUnit test rule, and
>>> its
>>> usages in the codebase for that.
>>> The rule launches a Flink MiniCluster within the same JVM, and
>>> submission to
>>> the mini cluster resembles how it would be submitting to an actual Flink
>>> cluster, so you would already be able to catch problems such as operator
>>> serialization errors.
>>>
>>> Cheers,
>>> Gordon
>>>
>>>
>>>
>>> --
>>> Sent from:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>>
>>
>>
>> --
>> *Laurent Exsteens*
>> Data Engineer
>> (M) +32 (0) 486 20 48 36
>>
>> *EURA NOVA*
>>
>> Rue Emile Francqui, 4
>>
>> 1435 Mont-Saint-Guibert
>>
>> (T) +32 10 75 02 00
>>
>> *euranova.eu <http://euranova.eu/>*
>>
>> *research.euranova.eu* <http://research.euranova.eu/>
>>
>> ♻ Be green, keep it on the screen
>
>
--
*Laurent Exsteens*
Data Engineer
(M) +32 (0) 486 20 48 36
*EURA NOVA*
Rue Emile Francqui, 4
1435 Mont-Saint-Guibert
(T) +32 10 75 02 00
*euranova.eu <http://euranova.eu/>*
*research.euranova.eu* <http://research.euranova.eu/>
--
♻ Be green, keep it on the screen
Re: Correct way to e2e test a Flink application?
Posted by Robert Metzger <rm...@apache.org>.
Hey Laurent,
Flink developed an internal framework for executing end to end tests from
Java. Here's an example of such a test:
https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-metrics-availability-test/src/test/java/org/pache/flink/metrics/tests/MetricsAvailabilityITCase.java#L98
This should download and set up a real, distributed (on your machine, but
in different processes) Flink cluster.
Best,
Robert
On Tue, Mar 31, 2020 at 10:48 AM Laurent Exsteens <
laurent.exsteens@euranova.eu> wrote:
> Hi Tzu-Li,
>
> thanks a lot for your answer. I will try this!
>
> However, I was looking for something that does fully simulate a Flink
> cluster, including job-manager to task manager serialization issues and
> full isolation of the task managers (I guess in the MiniClusterResource, we
> are still on the same JVM, so static classes will still be available to all
> operators, which is different than on a real cluster).
>
> Cheers,
>
> Laurent.
>
> On Mon, 2 Mar 2020 at 13:24, Tzu-Li (Gordon) Tai <tz...@apache.org>
> wrote:
>
>> Hi Laurent,
>>
>> You can take a look at Flink's MiniClusterResource JUnit test rule, and
>> its
>> usages in the codebase for that.
>> The rule launches a Flink MiniCluster within the same JVM, and submission
>> to
>> the mini cluster resembles how it would be submitting to an actual Flink
>> cluster, so you would already be able to catch problems such as operator
>> serialization errors.
>>
>> Cheers,
>> Gordon
>>
>>
>>
>> --
>> Sent from:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>>
>
>
> --
> *Laurent Exsteens*
> Data Engineer
> (M) +32 (0) 486 20 48 36
>
> *EURA NOVA*
>
> Rue Emile Francqui, 4
>
> 1435 Mont-Saint-Guibert
>
> (T) +32 10 75 02 00
>
> *euranova.eu <http://euranova.eu/>*
>
> *research.euranova.eu* <http://research.euranova.eu/>
>
> ♻ Be green, keep it on the screen
Re: Correct way to e2e test a Flink application?
Posted by Laurent Exsteens <la...@euranova.eu>.
Hi Tzu-Li,
thanks a lot for your answer. I will try this!
However, I was looking for something that does fully simulate a Flink
cluster, including job-manager to task manager serialization issues and
full isolation of the task managers (I guess in the MiniClusterResource, we
are still on the same JVM, so static classes will still be available to all
operators, which is different than on a real cluster).
Cheers,
Laurent.
On Mon, 2 Mar 2020 at 13:24, Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:
> Hi Laurent,
>
> You can take a look at Flink's MiniClusterResource JUnit test rule, and its
> usages in the codebase for that.
> The rule launches a Flink MiniCluster within the same JVM, and submission
> to
> the mini cluster resembles how it would be submitting to an actual Flink
> cluster, so you would already be able to catch problems such as operator
> serialization errors.
>
> Cheers,
> Gordon
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
--
*Laurent Exsteens*
Data Engineer
(M) +32 (0) 486 20 48 36
*EURA NOVA*
Rue Emile Francqui, 4
1435 Mont-Saint-Guibert
(T) +32 10 75 02 00
*euranova.eu <http://euranova.eu/>*
*research.euranova.eu* <http://research.euranova.eu/>
--
♻ Be green, keep it on the screen
Re: Correct way to e2e test a Flink application?
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Laurent,
You can take a look at Flink's MiniClusterResource JUnit test rule, and its
usages in the codebase for that.
The rule launches a Flink MiniCluster within the same JVM, and submission to
the mini cluster resembles how it would be submitting to an actual Flink
cluster, so you would already be able to catch problems such as operator
serialization errors.
Cheers,
Gordon
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/