You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Long Nguyễn <lo...@gmail.com> on 2021/11/05 04:20:16 UTC
How to write unit test for stateful operators in Pyflink apps
Hi.
I'm using Pyflink and Table APIs to create a window with Python.
I have just read the Unit Testing Stateful or Timely UDFs & Custom Operators
<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/testing/#unit-testing-stateful-or-timely-udfs--custom-operators>
section
in Flink's docs and it seems that the best way is to use the test harnesses.
However, test harnesses are only available with apps written in Java and I
have not found anything similar for Python yet.
So what is the proper way to unit test stateful operators, i.e. window, in
a Pyflink app at this moment?
Thanks in advance.
--
------------------------------------------------------------
--------------------------------------------------
Nguyen Dich Long,
School of Information and Communication Technology (SoICT -
https://www.soict.hust.edu.vn)
Hanoi University of Science and Technology (https://www.hust.edu.vn)
601, B1 Building - No 1, Dai Co Viet Street, Hai Ba Trung District, Ha Noi,
Vietnam
Tel: +84 (0)3.54.41.76.76
Email: long.nd162525@sis.hust.edu.vn; longnguyen25111998@gmail.com
Re: How to write unit test for stateful operators in Pyflink apps
Posted by Long Nguyễn <lo...@gmail.com>.
Hi Dian,
I got it.
A few days ago, I also found some test cases implemented with Python here
<https://github.com/apache/flink/tree/master/flink-python/pyflink/table/tests>
in
Flink's official repository. I took a look at them and it seems like many
internal functions are used and since those are not documented, it is very
hard to follow.
I think E2E stuff implemented with Python's unittest
<https://docs.python.org/3/library/unittest.html> or Pytest
<https://pytest.org/> is the most appropriate approach at this point, too.
Thank you guys for helping me. I really appreciate it.
Best,
Long
On Mon, Nov 8, 2021 at 8:25 AM Dian Fu <di...@gmail.com> wrote:
> Hi Long,
>
> I agree with Fabian that currently you have to test it with a e2e job.
> There are still no such test harnesses for PyFlink jobs.
>
> Regards,
> Dian
>
> On Fri, Nov 5, 2021 at 5:22 PM Long Nguyễn <lo...@gmail.com>
> wrote:
>
>> Thanks, Fabian. I'll check it out.
>>
>> Hope that Dian can also give me some advice.
>>
>> Best,
>> Long
>>
>> On Fri, Nov 5, 2021 at 3:48 PM Fabian Paul <fa...@ververica.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Since you want to use Table API you probably can write a more high-level
>>> test around executing the complete program. A good examples are the
>>> pyflink example programs [1].
>>>
>>> I also could not find something similar to the testing harness from
>>> Java. I cced Dian maybe he knows more about it.
>>>
>>> Best,
>>> Fabian
>>>
>>> [1]
>>> https://github.com/apache/flink/blob/3a561323f7a81990a4c239a886f8c6c404d92f0a/flink-python/pyflink/examples/table/word_count.py#L118
>>
>>
>>
>> --
>> ------------------------------------------------------------
>> --------------------------------------------------
>> Nguyen Dich Long,
>> School of Information and Communication Technology (SoICT -
>> https://www.soict.hust.edu.vn)
>> Hanoi University of Science and Technology (https://www.hust.edu.vn)
>> 601, B1 Building - No 1, Dai Co Viet Street, Hai Ba Trung District, Ha
>> Noi, Vietnam
>> Tel: +84 (0)3.54.41.76.76
>> Email: long.nd162525@sis.hust.edu.vn; longnguyen25111998@gmail.com
>>
>> --
>> <lo...@gmail.com>
>> <lo...@gmail.com>
>> ------------------------------------------------------------
>> --------------------------------------------------
>> Nguyen Dich Long,
>> School of Information and Communication Technology (SoICT -
>> <lo...@gmail.com>https://www.soict.hust.edu.vn)
>> Hanoi University of Science and Technology (https://www.hust.edu.vn)
>> 601, B1 Building - No 1, Dai Co Viet Street, Hai Ba Trung District, Ha
>> Noi, Vietnam
>> Tel: +84 (0)3.54.41.76.76
>> Email: long.nd162525@sis.hust.edu.vn; longnguyen25111998@gmail.com
>>
>>
Re: How to write unit test for stateful operators in Pyflink apps
Posted by Dian Fu <di...@gmail.com>.
Hi Long,
I agree with Fabian that currently you have to test it with a e2e job.
There are still no such test harnesses for PyFlink jobs.
Regards,
Dian
On Fri, Nov 5, 2021 at 5:22 PM Long Nguyễn <lo...@gmail.com>
wrote:
> Thanks, Fabian. I'll check it out.
>
> Hope that Dian can also give me some advice.
>
> Best,
> Long
>
> On Fri, Nov 5, 2021 at 3:48 PM Fabian Paul <fa...@ververica.com>
> wrote:
>
>> Hi,
>>
>> Since you want to use Table API you probably can write a more high-level
>> test around executing the complete program. A good examples are the
>> pyflink example programs [1].
>>
>> I also could not find something similar to the testing harness from Java.
>> I cced Dian maybe he knows more about it.
>>
>> Best,
>> Fabian
>>
>> [1]
>> https://github.com/apache/flink/blob/3a561323f7a81990a4c239a886f8c6c404d92f0a/flink-python/pyflink/examples/table/word_count.py#L118
>
>
>
> --
> ------------------------------------------------------------
> --------------------------------------------------
> Nguyen Dich Long,
> School of Information and Communication Technology (SoICT -
> https://www.soict.hust.edu.vn)
> Hanoi University of Science and Technology (https://www.hust.edu.vn)
> 601, B1 Building - No 1, Dai Co Viet Street, Hai Ba Trung District, Ha
> Noi, Vietnam
> Tel: +84 (0)3.54.41.76.76
> Email: long.nd162525@sis.hust.edu.vn; longnguyen25111998@gmail.com
>
Re: How to write unit test for stateful operators in Pyflink apps
Posted by Long Nguyễn <lo...@gmail.com>.
Thanks, Fabian. I'll check it out.
Hope that Dian can also give me some advice.
Best,
Long
On Fri, Nov 5, 2021 at 3:48 PM Fabian Paul <fa...@ververica.com> wrote:
> Hi,
>
> Since you want to use Table API you probably can write a more high-level
> test around executing the complete program. A good examples are the
> pyflink example programs [1].
>
> I also could not find something similar to the testing harness from Java.
> I cced Dian maybe he knows more about it.
>
> Best,
> Fabian
>
> [1]
> https://github.com/apache/flink/blob/3a561323f7a81990a4c239a886f8c6c404d92f0a/flink-python/pyflink/examples/table/word_count.py#L118
--
------------------------------------------------------------
--------------------------------------------------
Nguyen Dich Long,
School of Information and Communication Technology (SoICT -
https://www.soict.hust.edu.vn)
Hanoi University of Science and Technology (https://www.hust.edu.vn)
601, B1 Building - No 1, Dai Co Viet Street, Hai Ba Trung District, Ha Noi,
Vietnam
Tel: +84 (0)3.54.41.76.76
Email: long.nd162525@sis.hust.edu.vn; longnguyen25111998@gmail.com
Re: How to write unit test for stateful operators in Pyflink apps
Posted by Fabian Paul <fa...@ververica.com>.
Hi,
Since you want to use Table API you probably can write a more high-level test around executing the complete program. A good examples are the
pyflink example programs [1].
I also could not find something similar to the testing harness from Java. I cced Dian maybe he knows more about it.
Best,
Fabian
[1] https://github.com/apache/flink/blob/3a561323f7a81990a4c239a886f8c6c404d92f0a/flink-python/pyflink/examples/table/word_count.py#L118