You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Joe Malt <jm...@yelp.com> on 2018/08/24 16:55:19 UTC
Re: [External] Re: How to do test in Flink?
Hi Chang,
A time-saving tip for finding which library contains a class: go to
https://search.maven.org/
and enter fc: followed by the fully-qualified name of the class. You should
get the library as a search result.
In this case for example, you'd search for
fc:org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder
Best,
Joe Malt
Engineering Intern, Stream Processing
Yelp Inc.
On Fri, Aug 24, 2018 at 4:50 AM, Chang Liu <fl...@gmail.com> wrote:
> No worries, I found it here:
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-runtime_${scala.binary.version}</artifactId>
> <version>${flink.version}</version>
> <type>test-jar</type>
> <scope>test</scope>
> </dependency>
>
>
> Best regards/祝好,
>
> Chang Liu 刘畅
>
>
>
> On Fri, Aug 24, 2018 at 1:16 PM Chang Liu <fl...@gmail.com> wrote:
>
>> Hi Hequn,
>>
>> I have added the following dependencies:
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
>> <version>${flink.version}</version>
>> <type>test-jar</type>
>> <scope>test</scope>
>> </dependency>
>> <dependency>
>> <groupId>org.mockito</groupId>
>> <artifactId>mockito-core</artifactId>
>> <version>2.21.0</version>
>> <scope>test</scope>
>> </dependency>
>>
>>
>> But got the exception: java.lang.NoClassDefFoundError:
>> org/apache/flink/runtime/operators/testutils/MockEnvironmentBuilder
>>
>> Do you know which library contains this class? Thanks :)
>>
>> Best regards/祝好,
>>
>> Chang Liu 刘畅
>> DevOps Engineer
>> WB TECH / Cyber Crime Prevention Team
>>
>> Mobile: +31(0)687859981
>> Email: fluency.03@gmail.com & Chang.Liu2@ing.nl
>>
>>
>>
>> On Mon, Aug 13, 2018 at 1:42 PM Hequn Cheng <ch...@gmail.com> wrote:
>>
>>> Hi Change,
>>>
>>> Try
>>> <dependency>
>>> <groupId>org.apache.flink</groupId>
>>> <artifactId>flink-streaming-java_2.11</artifactId>
>>> <version>${flink.version}</version>
>>> <type>test-jar</type>
>>> <scope>test</scope>
>>> </dependency>
>>> .
>>>
>>> On Mon, Aug 13, 2018 at 6:42 PM, Chang Liu <fl...@gmail.com> wrote:
>>>
>>>> And another question: which library should I include in order to use
>>>> these harnesses? I do have this flink-test-utils_2.11 in my pom, but I
>>>> cannot find the harnesses.
>>>>
>>>> I also have the following in my pom:
>>>>
>>>> - flink-core
>>>> - flink-clients_2.11
>>>> - flink-scala_2.11
>>>> - flink-streaming-java_2.11
>>>> - flink-streaming-java_2.11
>>>> - flink-connector-kafka-0.11_2.11
>>>>
>>>>
>>>> Best regards/祝好,
>>>>
>>>> Chang Liu 刘畅
>>>>
>>>>
>>>> On 13 Aug 2018, at 04:01, Hequn Cheng <ch...@gmail.com> wrote:
>>>>
>>>> Hi Chang,
>>>>
>>>> There are some harness tests which can be used to test your function.
>>>> It is also a common way to test function or operator in flink internal
>>>> tests. Currently, the harness classes mainly include:
>>>>
>>>> - KeyedOneInputStreamOperatorTestHarness
>>>> - KeyedTwoInputStreamOperatorTestHarness
>>>> - OneInputStreamOperatorTestHarness
>>>> - TwoInputStreamOperatorTestHarness
>>>>
>>>> You can take a look at the source code of these classes.
>>>>
>>>> To be more specific, you can take a look at the
>>>> testSlidingEventTimeWindowsApply[1], in which the RichSumReducer
>>>> window function has been tested.
>>>>
>>>> Best, Hequn
>>>>
>>>> [1] https://github.com/apache/flink/blob/master/flink-
>>>> streaming-java/src/test/java/org/apache/flink/streaming/
>>>> runtime/operators/windowing/WindowOperatorTest.java#L213
>>>>
>>>>
>>>> On Mon, Aug 13, 2018 at 7:10 AM, Chang Liu <fl...@gmail.com>
>>>> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I have some questions regarding testing in Flink. The more general
>>>>> question is: is there any guideline, template, or best practices that we
>>>>> can follow if we want to test our flink code (more in scala)?
>>>>>
>>>>> I know there is this page: https://ci.apache.org/
>>>>> projects/flink/flink-docs-release-1.6/dev/stream/testing.html but not
>>>>> so much written there. And I also did not find a more comprehensive
>>>>> documentation of this library: flink-test-utils_2.11.
>>>>>
>>>>> One detailed question: how do you test this WindowFunction below? The
>>>>> return type is Unit right? We cannot do unit test on like, like how the
>>>>> ReduceFunction was tested in the example link above. Then we only have the
>>>>> option of doing integration testing on it?
>>>>> <code.png>
>>>>>
>>>>>
>>>>> Your ideas would be very helpful :) Thanks in advance !
>>>>>
>>>>> Best regards/祝好,
>>>>>
>>>>> Chang Liu 刘畅
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
Re: [External] Re: How to do test in Flink?
Posted by Chang Liu <fl...@gmail.com>.
Thanks Joe!
Best regards/祝好,
Chang Liu 刘畅
On Fri, Aug 24, 2018 at 6:55 PM Joe Malt <jm...@yelp.com> wrote:
> Hi Chang,
>
> A time-saving tip for finding which library contains a class: go to
> https://search.maven.org/
> and enter fc: followed by the fully-qualified name of the class. You
> should get the library as a search result.
>
> In this case for example, you'd search for
> fc:org.apache.flink.runtime.operators.testutils.MockEnvironmentBuilder
>
> Best,
>
> Joe Malt
> Engineering Intern, Stream Processing
> Yelp Inc.
>
> On Fri, Aug 24, 2018 at 4:50 AM, Chang Liu <fl...@gmail.com> wrote:
>
>> No worries, I found it here:
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-runtime_${scala.binary.version}</artifactId>
>> <version>${flink.version}</version>
>> <type>test-jar</type>
>> <scope>test</scope>
>> </dependency>
>>
>>
>> Best regards/祝好,
>>
>> Chang Liu 刘畅
>>
>>
>>
>> On Fri, Aug 24, 2018 at 1:16 PM Chang Liu <fl...@gmail.com> wrote:
>>
>>> Hi Hequn,
>>>
>>> I have added the following dependencies:
>>>
>>> <dependency>
>>> <groupId>org.apache.flink</groupId>
>>> <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
>>> <version>${flink.version}</version>
>>> <type>test-jar</type>
>>> <scope>test</scope>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.mockito</groupId>
>>> <artifactId>mockito-core</artifactId>
>>> <version>2.21.0</version>
>>> <scope>test</scope>
>>> </dependency>
>>>
>>>
>>> But got the exception: java.lang.NoClassDefFoundError:
>>> org/apache/flink/runtime/operators/testutils/MockEnvironmentBuilder
>>>
>>> Do you know which library contains this class? Thanks :)
>>>
>>> Best regards/祝好,
>>>
>>> Chang Liu 刘畅
>>> DevOps Engineer
>>> WB TECH / Cyber Crime Prevention Team
>>>
>>> Mobile: +31(0)687859981
>>> Email: fluency.03@gmail.com & Chang.Liu2@ing.nl
>>>
>>>
>>>
>>> On Mon, Aug 13, 2018 at 1:42 PM Hequn Cheng <ch...@gmail.com>
>>> wrote:
>>>
>>>> Hi Change,
>>>>
>>>> Try
>>>> <dependency>
>>>> <groupId>org.apache.flink</groupId>
>>>> <artifactId>flink-streaming-java_2.11</artifactId>
>>>> <version>${flink.version}</version>
>>>> <type>test-jar</type>
>>>> <scope>test</scope>
>>>> </dependency>
>>>> .
>>>>
>>>> On Mon, Aug 13, 2018 at 6:42 PM, Chang Liu <fl...@gmail.com>
>>>> wrote:
>>>>
>>>>> And another question: which library should I include in order to use
>>>>> these harnesses? I do have this flink-test-utils_2.11 in my pom, but I
>>>>> cannot find the harnesses.
>>>>>
>>>>> I also have the following in my pom:
>>>>>
>>>>> - flink-core
>>>>> - flink-clients_2.11
>>>>> - flink-scala_2.11
>>>>> - flink-streaming-java_2.11
>>>>> - flink-streaming-java_2.11
>>>>> - flink-connector-kafka-0.11_2.11
>>>>>
>>>>>
>>>>> Best regards/祝好,
>>>>>
>>>>> Chang Liu 刘畅
>>>>>
>>>>>
>>>>> On 13 Aug 2018, at 04:01, Hequn Cheng <ch...@gmail.com> wrote:
>>>>>
>>>>> Hi Chang,
>>>>>
>>>>> There are some harness tests which can be used to test your function.
>>>>> It is also a common way to test function or operator in flink internal
>>>>> tests. Currently, the harness classes mainly include:
>>>>>
>>>>> - KeyedOneInputStreamOperatorTestHarness
>>>>> - KeyedTwoInputStreamOperatorTestHarness
>>>>> - OneInputStreamOperatorTestHarness
>>>>> - TwoInputStreamOperatorTestHarness
>>>>>
>>>>> You can take a look at the source code of these classes.
>>>>>
>>>>> To be more specific, you can take a look at
>>>>> the testSlidingEventTimeWindowsApply[1], in which the RichSumReducer window
>>>>> function has been tested.
>>>>>
>>>>> Best, Hequn
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java#L213
>>>>>
>>>>>
>>>>> On Mon, Aug 13, 2018 at 7:10 AM, Chang Liu <fl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I have some questions regarding testing in Flink. The more general
>>>>>> question is: is there any guideline, template, or best practices that we
>>>>>> can follow if we want to test our flink code (more in scala)?
>>>>>>
>>>>>> I know there is this page:
>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/testing.html but
>>>>>> not so much written there. And I also did not find a more comprehensive
>>>>>> documentation of this library: flink-test-utils_2.11.
>>>>>>
>>>>>> One detailed question: how do you test this WindowFunction below? The
>>>>>> return type is Unit right? We cannot do unit test on like, like how the
>>>>>> ReduceFunction was tested in the example link above. Then we only have the
>>>>>> option of doing integration testing on it?
>>>>>> <code.png>
>>>>>>
>>>>>>
>>>>>> Your ideas would be very helpful :) Thanks in advance !
>>>>>>
>>>>>> Best regards/祝好,
>>>>>>
>>>>>> Chang Liu 刘畅
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>