You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Nicos Maris <ni...@gmail.com> on 2018/08/28 11:35:24 UTC
test windows
Hi all,
How can I test in Java any streaming job that has a time window?
best,
Nicos
Re: test windows
Posted by Hequn Cheng <ch...@gmail.com>.
Hi Nicos,
Do you want an IT test to test the job, or just a UT to test the window
operator?
If you want to test the job, there are examples here:
1. WordCountITCase.java
<https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/example/java/WordCountITCase.java>
2. WindowFunctionITCase.scala
<https://github.com/apache/flink/blob/master/flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/WindowFunctionITCase.scala>
Also, there are UT tests using test harness to test a window operator:
1. WindowOperatorTest.java
<https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java>
Best, Hequn
On Thu, Aug 30, 2018 at 9:04 PM Nicos Maris <ni...@gmail.com> wrote:
> Can the OneInputStreamOperatorTestHarness
> <https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java>
> be used to test pipelines with time windows?
>
> Can someone explain this test harness preferably with a tiny example?
>
> I see here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/testing.html#testing-checkpointing-and-state-handling>
> that "it is currently not a part of public API and can be subject to
> change", but I assume it is the only thing I can get from flink as regards
> automated testing of pipelines with time windows.
>
> On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <ni...@gmail.com> wrote:
>
>> Hi Zhengwen,
>>
>>
>> I have tested my job manually (both by submitting a job and through
>> execute()) and I am trying to write a test.
>>
>> The following project states that it has the feature "Test stream
>> windowing with timestamped input" but I do not want to rely on a project
>> other than flink.
>>
>> https://github.com/ottogroup/flink-spector
>>
>> The following folder doesn't seam to demonstrate such capability for
>> testing.
>>
>>
>> https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test
>>
>>
>>
>> On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <ah...@gmail.com> wrote:
>>
>>> Hi Nicos,
>>> Under the flink-example module, there are many examples, including batch
>>> and streaming. You could build the project from the source, this way you
>>> could found many jars under the target directory. You can submit these jars
>>> to the Flink cluster. Also, you could run these examples directly from your
>>> IDE by standalone mode.
>>>
>>> best,
>>> Zhengwen
>>>
>>> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>> How can I test in Java any streaming job that has a time window?
>>>>
>>>>
>>>> best,
>>>> Nicos
>>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>>
>>> *_________________________________________Zhu ZhengwenState Key
>>> Laboratory For Novel Software TechnologyDept. of Computer Sci. & Tech.,
>>> Nanjing University*
>>> *9, Hankou Road, Nanjing, China.*
>>>
>>
Re: test windows
Posted by Hequn Cheng <ch...@gmail.com>.
Hi Nicos,
I think this is what you want. WindowFoldITCase
<https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/streaming/runtime/WindowFoldITCase.java>.
:-)
Flink is a open source project, you can checkout the source code from git
and search the tests in it. For example, you can search the class by name
that contains 'window'. There are a lot of other test cases which I think
will help you.
Best, Hequn
On Sun, Sep 9, 2018 at 10:29 AM Nicos Maris <ni...@gmail.com> wrote:
> Hi all,
>
>
> It took me some time to reply.
>
> I am focusing on testing my job, not on unit tests and the SourceFunction
> as in WindowFunctionITCase.scala
> <https://github.com/apache/flink/blob/master/flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/WindowFunctionITCase.scala> has
> helped me a lot :)
>
>
> Is there any test with keyed windows?
>
> I am writing in java with POJOs instead of tuples.
>
> On Fri, Aug 31, 2018 at 3:43 PM David Anderson <da...@data-artisans.com>
> wrote:
>
>> The flink training exercises have a simpler example of using a
>> TwoInputStreamOperatorTestHarness from outside of the Flink code base that
>> you can refer to. The two input test harness is more or less the same as
>> the one input test harness.
>>
>>
>> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/test/java/com/dataartisans/flinktraining/exercises/datastream_java/process/EventTimeJoinTest.java
>>
>> You'll need these dependencies:
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-test-utils-junit</artifactId>
>> <version>${flink.version}</version>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-streaming-java_2.11</artifactId>
>> <version>${flink.version}</version>
>> <scope>test</scope>
>> <type>test-jar</type>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.mockito</groupId>
>> <artifactId>mockito-all</artifactId>
>> <version>1.10.19</version>
>> <type>jar</type>
>> <scope>test</scope>
>> </dependency>
>>
>> <dependency>
>> <groupId>org.apache.flink</groupId>
>> <artifactId>flink-runtime_2.11</artifactId>
>> <version>${flink.version}</version>
>> <scope>test</scope>
>> <type>test-jar</type>
>> </dependency>
>>
>> On Thu, Aug 30, 2018 at 3:04 PM Nicos Maris <ni...@gmail.com>
>> wrote:
>> >
>> > Can the OneInputStreamOperatorTestHarness be used to test pipelines
>> with time windows?
>> >
>> > Can someone explain this test harness preferably with a tiny example?
>> >
>> > I see here that "it is currently not a part of public API and can be
>> subject to change", but I assume it is the only thing I can get from flink
>> as regards automated testing of pipelines with time windows.
>> >
>> > On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <ni...@gmail.com>
>> wrote:
>> >>
>> >> Hi Zhengwen,
>> >>
>> >>
>> >> I have tested my job manually (both by submitting a job and through
>> execute()) and I am trying to write a test.
>> >>
>> >> The following project states that it has the feature "Test stream
>> windowing with timestamped input" but I do not want to rely on a project
>> other than flink.
>> >>
>> >> https://github.com/ottogroup/flink-spector
>> >>
>> >> The following folder doesn't seam to demonstrate such capability for
>> testing.
>> >>
>> >>
>> https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test
>> >>
>> >>
>> >>
>> >> On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <ah...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi Nicos,
>> >>> Under the flink-example module, there are many examples, including
>> batch and streaming. You could build the project from the source, this way
>> you could found many jars under the target directory. You can submit these
>> jars to the Flink cluster. Also, you could run these examples directly from
>> your IDE by standalone mode.
>> >>>
>> >>> best,
>> >>> Zhengwen
>> >>>
>> >>> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com>
>> wrote:
>> >>>>
>> >>>> Hi all,
>> >>>>
>> >>>>
>> >>>> How can I test in Java any streaming job that has a time window?
>> >>>>
>> >>>>
>> >>>> best,
>> >>>> Nicos
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> _________________________________________
>> >>>
>> >>> Zhu Zhengwen
>> >>> State Key Laboratory For Novel Software Technology
>> >>> Dept. of Computer Sci. & Tech., Nanjing University
>> >>> 9, Hankou Road, Nanjing, China.
>>
>>
>>
>> --
>> David Anderson | Training Coordinator | data Artisans
>> --
>> Join Flink Forward - The Apache Flink Conference
>> Stream Processing | Event Driven | Real Time
>>
>
Re: test windows
Posted by Nicos Maris <ni...@gmail.com>.
Hi all,
It took me some time to reply.
I am focusing on testing my job, not on unit tests and the SourceFunction
as in WindowFunctionITCase.scala
<https://github.com/apache/flink/blob/master/flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/WindowFunctionITCase.scala>
has
helped me a lot :)
Is there any test with keyed windows?
I am writing in java with POJOs instead of tuples.
On Fri, Aug 31, 2018 at 3:43 PM David Anderson <da...@data-artisans.com>
wrote:
> The flink training exercises have a simpler example of using a
> TwoInputStreamOperatorTestHarness from outside of the Flink code base that
> you can refer to. The two input test harness is more or less the same as
> the one input test harness.
>
>
> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/test/java/com/dataartisans/flinktraining/exercises/datastream_java/process/EventTimeJoinTest.java
>
> You'll need these dependencies:
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-test-utils-junit</artifactId>
> <version>${flink.version}</version>
> </dependency>
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-streaming-java_2.11</artifactId>
> <version>${flink.version}</version>
> <scope>test</scope>
> <type>test-jar</type>
> </dependency>
>
> <dependency>
> <groupId>org.mockito</groupId>
> <artifactId>mockito-all</artifactId>
> <version>1.10.19</version>
> <type>jar</type>
> <scope>test</scope>
> </dependency>
>
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-runtime_2.11</artifactId>
> <version>${flink.version}</version>
> <scope>test</scope>
> <type>test-jar</type>
> </dependency>
>
> On Thu, Aug 30, 2018 at 3:04 PM Nicos Maris <ni...@gmail.com> wrote:
> >
> > Can the OneInputStreamOperatorTestHarness be used to test pipelines with
> time windows?
> >
> > Can someone explain this test harness preferably with a tiny example?
> >
> > I see here that "it is currently not a part of public API and can be
> subject to change", but I assume it is the only thing I can get from flink
> as regards automated testing of pipelines with time windows.
> >
> > On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <ni...@gmail.com>
> wrote:
> >>
> >> Hi Zhengwen,
> >>
> >>
> >> I have tested my job manually (both by submitting a job and through
> execute()) and I am trying to write a test.
> >>
> >> The following project states that it has the feature "Test stream
> windowing with timestamped input" but I do not want to rely on a project
> other than flink.
> >>
> >> https://github.com/ottogroup/flink-spector
> >>
> >> The following folder doesn't seam to demonstrate such capability for
> testing.
> >>
> >>
> https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test
> >>
> >>
> >>
> >> On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <ah...@gmail.com> wrote:
> >>>
> >>> Hi Nicos,
> >>> Under the flink-example module, there are many examples, including
> batch and streaming. You could build the project from the source, this way
> you could found many jars under the target directory. You can submit these
> jars to the Flink cluster. Also, you could run these examples directly from
> your IDE by standalone mode.
> >>>
> >>> best,
> >>> Zhengwen
> >>>
> >>> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com>
> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>>
> >>>> How can I test in Java any streaming job that has a time window?
> >>>>
> >>>>
> >>>> best,
> >>>> Nicos
> >>>
> >>>
> >>>
> >>> --
> >>> _________________________________________
> >>>
> >>> Zhu Zhengwen
> >>> State Key Laboratory For Novel Software Technology
> >>> Dept. of Computer Sci. & Tech., Nanjing University
> >>> 9, Hankou Road, Nanjing, China.
>
>
>
> --
> David Anderson | Training Coordinator | data Artisans
> --
> Join Flink Forward - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
>
Re: test windows
Posted by David Anderson <da...@data-artisans.com>.
The flink training exercises have a simpler example of using a
TwoInputStreamOperatorTestHarness from outside of the Flink code base that
you can refer to. The two input test harness is more or less the same as
the one input test harness.
https://github.com/dataArtisans/flink-training-exercises/blob/master/src/test/java/com/dataartisans/flinktraining/exercises/datastream_java/process/EventTimeJoinTest.java
You'll need these dependencies:
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-test-utils-junit</artifactId>
<version>${flink.version}</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_2.11</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.10.19</version>
<type>jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-runtime_2.11</artifactId>
<version>${flink.version}</version>
<scope>test</scope>
<type>test-jar</type>
</dependency>
On Thu, Aug 30, 2018 at 3:04 PM Nicos Maris <ni...@gmail.com> wrote:
>
> Can the OneInputStreamOperatorTestHarness be used to test pipelines with
time windows?
>
> Can someone explain this test harness preferably with a tiny example?
>
> I see here that "it is currently not a part of public API and can be
subject to change", but I assume it is the only thing I can get from flink
as regards automated testing of pipelines with time windows.
>
> On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <ni...@gmail.com> wrote:
>>
>> Hi Zhengwen,
>>
>>
>> I have tested my job manually (both by submitting a job and through
execute()) and I am trying to write a test.
>>
>> The following project states that it has the feature "Test stream
windowing with timestamped input" but I do not want to rely on a project
other than flink.
>>
>> https://github.com/ottogroup/flink-spector
>>
>> The following folder doesn't seam to demonstrate such capability for
testing.
>>
>>
https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test
>>
>>
>>
>> On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <ah...@gmail.com> wrote:
>>>
>>> Hi Nicos,
>>> Under the flink-example module, there are many examples, including
batch and streaming. You could build the project from the source, this way
you could found many jars under the target directory. You can submit these
jars to the Flink cluster. Also, you could run these examples directly from
your IDE by standalone mode.
>>>
>>> best,
>>> Zhengwen
>>>
>>> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com>
wrote:
>>>>
>>>> Hi all,
>>>>
>>>>
>>>> How can I test in Java any streaming job that has a time window?
>>>>
>>>>
>>>> best,
>>>> Nicos
>>>
>>>
>>>
>>> --
>>> _________________________________________
>>>
>>> Zhu Zhengwen
>>> State Key Laboratory For Novel Software Technology
>>> Dept. of Computer Sci. & Tech., Nanjing University
>>> 9, Hankou Road, Nanjing, China.
--
David Anderson | Training Coordinator | data Artisans
--
Join Flink Forward - The Apache Flink Conference
Stream Processing | Event Driven | Real Time
Re: test windows
Posted by Nicos Maris <ni...@gmail.com>.
Can the OneInputStreamOperatorTestHarness
<https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperatorTest.java>
be used to test pipelines with time windows?
Can someone explain this test harness preferably with a tiny example?
I see here
<https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/testing.html#testing-checkpointing-and-state-handling>
that "it is currently not a part of public API and can be subject to
change", but I assume it is the only thing I can get from flink as regards
automated testing of pipelines with time windows.
On Tue, Aug 28, 2018 at 3:12 PM Nicos Maris <ni...@gmail.com> wrote:
> Hi Zhengwen,
>
>
> I have tested my job manually (both by submitting a job and through
> execute()) and I am trying to write a test.
>
> The following project states that it has the feature "Test stream
> windowing with timestamped input" but I do not want to rely on a project
> other than flink.
>
> https://github.com/ottogroup/flink-spector
>
> The following folder doesn't seam to demonstrate such capability for
> testing.
>
>
> https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test
>
>
>
> On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <ah...@gmail.com> wrote:
>
>> Hi Nicos,
>> Under the flink-example module, there are many examples, including batch
>> and streaming. You could build the project from the source, this way you
>> could found many jars under the target directory. You can submit these jars
>> to the Flink cluster. Also, you could run these examples directly from your
>> IDE by standalone mode.
>>
>> best,
>> Zhengwen
>>
>> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>>
>>> How can I test in Java any streaming job that has a time window?
>>>
>>>
>>> best,
>>> Nicos
>>>
>>
>>
>> --
>>
>>
>>
>>
>> *_________________________________________Zhu ZhengwenState Key
>> Laboratory For Novel Software TechnologyDept. of Computer Sci. & Tech.,
>> Nanjing University*
>> *9, Hankou Road, Nanjing, China.*
>>
>
Re: test windows
Posted by Nicos Maris <ni...@gmail.com>.
Hi Zhengwen,
I have tested my job manually (both by submitting a job and through
execute()) and I am trying to write a test.
The following project states that it has the feature "Test stream windowing
with timestamped input" but I do not want to rely on a project other than
flink.
https://github.com/ottogroup/flink-spector
The following folder doesn't seam to demonstrate such capability for
testing.
https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming/src/test/java/org/apache/flink/streaming/test
On Tue, Aug 28, 2018 at 2:55 PM ZhengWen ZHU <ah...@gmail.com> wrote:
> Hi Nicos,
> Under the flink-example module, there are many examples, including batch
> and streaming. You could build the project from the source, this way you
> could found many jars under the target directory. You can submit these jars
> to the Flink cluster. Also, you could run these examples directly from your
> IDE by standalone mode.
>
> best,
> Zhengwen
>
> On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com> wrote:
>
>> Hi all,
>>
>>
>> How can I test in Java any streaming job that has a time window?
>>
>>
>> best,
>> Nicos
>>
>
>
> --
>
>
>
>
> *_________________________________________Zhu ZhengwenState Key Laboratory
> For Novel Software TechnologyDept. of Computer Sci. & Tech., Nanjing
> University*
> *9, Hankou Road, Nanjing, China.*
>
Re: test windows
Posted by ZhengWen ZHU <ah...@gmail.com>.
Hi Nicos,
Under the flink-example module, there are many examples, including batch
and streaming. You could build the project from the source, this way you
could found many jars under the target directory. You can submit these jars
to the Flink cluster. Also, you could run these examples directly from your
IDE by standalone mode.
best,
Zhengwen
On Tue, Aug 28, 2018 at 7:35 PM Nicos Maris <ni...@gmail.com> wrote:
> Hi all,
>
>
> How can I test in Java any streaming job that has a time window?
>
>
> best,
> Nicos
>
--
*_________________________________________Zhu ZhengwenState Key Laboratory
For Novel Software TechnologyDept. of Computer Sci. & Tech., Nanjing
University*
*9, Hankou Road, Nanjing, China.*