test (harness and minicluster)

[Martin Frank Hansen <ma...@berlingskemedia.dk>]

Hi,

I am trying to make a test-suite for our flink cluster using harness and
minicluster. As we are using processWindowFunctions in our pipeline we need
some good ways of validating these functions. To my surprise
processWindowFunctions are neither supported by test-harness or minicluster
setups, so does anyone know if it will be supported in the future? (Or if
it is supported how should I use it?)

Furthermore does anyone have some good ideas for a test-setup for
processWindowFunctions?

best regards


Martin Frank Hansen

Data Engineer

Re: test (harness and minicluster)

[Martin Frank Hansen <ma...@berlingskemedia.dk>]

Hi Till,
Thanks for your answer!
I will try the approach you suggested, hope I can make it work.

Best regards
martin

Den ons. 2. dec. 2020 kl. 17.03 skrev Till Rohrmann <tr...@apache.org>:

> Hi Martin,
>
> In general, Flink's MiniCluster should be able to run every complete Flink
> JobGraph. However, from what I read you are looking for a test harness for
> a processWindowFunction so that you can test this function in a more unit
> test style, right? What you can do is to use the
> OneInputStreamOperatorTestHarness and initialize it with a WindowOperator
> where you pass the ProcessWindowFunction to. That way you should be able to
> test the ProcessWindowFunction. Please also take a look
> at org.apache.flink.streaming.runtime.operators.windowing.WindowOperatorTest
> [1] for some examples how it could be done.
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java
>
> Cheers,
> Till
>
> On Wed, Dec 2, 2020 at 12:04 PM Martin Frank Hansen <
> mafh@berlingskemedia.dk> wrote:
>
>> Hi,
>>
>> I am trying to make a test-suite for our flink cluster using harness and
>> minicluster. As we are using processWindowFunctions in our pipeline we need
>> some good ways of validating these functions. To my surprise
>> processWindowFunctions are neither supported by test-harness or minicluster
>> setups, so does anyone know if it will be supported in the future? (Or if
>> it is supported how should I use it?)
>>
>> Furthermore does anyone have some good ideas for a test-setup for
>> processWindowFunctions?
>>
>> best regards
>>
>>
>> Martin Frank Hansen
>>
>> Data Engineer
>>
>>

-- 

Martin Frank Hansen

Data Engineer
Digital Service
M: +45 25 57 14 18
E: mafh@berlingskemedia.dk

Pilestræde 34 | DK-1147 København K | T: +45 33 75 75 75 |
berlingskemedia.dk

Re: test (harness and minicluster)

[Till Rohrmann <tr...@apache.org>]

Hi Martin,

In general, Flink's MiniCluster should be able to run every complete Flink
JobGraph. However, from what I read you are looking for a test harness for
a processWindowFunction so that you can test this function in a more unit
test style, right? What you can do is to use the
OneInputStreamOperatorTestHarness and initialize it with a WindowOperator
where you pass the ProcessWindowFunction to. That way you should be able to
test the ProcessWindowFunction. Please also take a look
at org.apache.flink.streaming.runtime.operators.windowing.WindowOperatorTest
[1] for some examples how it could be done.

[1]
https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java

Cheers,
Till

On Wed, Dec 2, 2020 at 12:04 PM Martin Frank Hansen <ma...@berlingskemedia.dk>
wrote:

> Hi,
>
> I am trying to make a test-suite for our flink cluster using harness and
> minicluster. As we are using processWindowFunctions in our pipeline we need
> some good ways of validating these functions. To my surprise
> processWindowFunctions are neither supported by test-harness or minicluster
> setups, so does anyone know if it will be supported in the future? (Or if
> it is supported how should I use it?)
>
> Furthermore does anyone have some good ideas for a test-setup for
> processWindowFunctions?
>
> best regards
>
>
> Martin Frank Hansen
>
> Data Engineer
>
>