You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Kevin Lam <ke...@shopify.com> on 2021/03/18 14:46:23 UTC

Python API + Unit Testing

Hi all,

I noticed there isn't much in the way of testing discussed in the Python
API docs
<https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/>
for Flink.

Does the community have any best-practices or recommendations on how
testing should be done with PyFlink?

Thanks!

Re: Python API + Unit Testing

Posted by Dian Fu <di...@gmail.com>.

Hi Kevin,

>> One approach I could see is using StreamingFileSinks, and validating the
output files for a bounded stream.

This is a good approach from my point of view.  Actually, the end to end
tests in Flink also takes this kind of approach. This doesn't only apply
for Python jobs, but also for Java / Scala / SQL jobs.

Regards,
Dian

On Fri, Mar 19, 2021 at 10:37 PM Kevin Lam <ke...@shopify.com> wrote:

> Hi Dian Fu,
>
> I meant testing in application development. When I'm developing a Pyflink
> Pipeline, are there any recommended approaches to testing the Flink
> application? For instance, how should we test applications end-to-end?
> Individual operators?
> I'm interested in the Datastream API.
>
> One approach I could see is using StreamingFileSinks, and validating the
> output files for a bounded stream.
>
> On Thu, Mar 18, 2021 at 10:04 PM Dian Fu <di...@gmail.com> wrote:
>
>> Hi,
>>
>> Do you mean how to run Python unit test? If so, you could refer to [1]
>> for more details.
>>
>> Regards,
>> Dian
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment
>>
>> 2021年3月18日 下午10:46，Kevin Lam <ke...@shopify.com> 写道：
>>
>> Hi all,
>>
>> I noticed there isn't much in the way of testing discussed in the Python
>> API docs
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/>
>> for Flink.
>>
>> Does the community have any best-practices or recommendations on how
>> testing should be done with PyFlink?
>>
>> Thanks!
>>
>>
>>

Re: Python API + Unit Testing

Posted by Kevin Lam <ke...@shopify.com>.

Hi Dian Fu,

I meant testing in application development. When I'm developing a Pyflink
Pipeline, are there any recommended approaches to testing the Flink
application? For instance, how should we test applications end-to-end?
Individual operators?
I'm interested in the Datastream API.

One approach I could see is using StreamingFileSinks, and validating the
output files for a bounded stream.

On Thu, Mar 18, 2021 at 10:04 PM Dian Fu <di...@gmail.com> wrote:

> Hi,
>
> Do you mean how to run Python unit test? If so, you could refer to [1] for
> more details.
>
> Regards,
> Dian
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment
>
> 2021年3月18日 下午10:46，Kevin Lam <ke...@shopify.com> 写道：
>
> Hi all,
>
> I noticed there isn't much in the way of testing discussed in the Python
> API docs
> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/>
> for Flink.
>
> Does the community have any best-practices or recommendations on how
> testing should be done with PyFlink?
>
> Thanks!
>
>
>

Re: Python API + Unit Testing

Posted by Dian Fu <di...@gmail.com>.

Hi,

Do you mean how to run Python unit test? If so, you could refer to [1] for more details.

Regards,
Dian

[1] https://cwiki.apache.org/confluence/display/FLINK/Setting+up+a+Flink+development+environment

> 2021年3月18日 下午10:46，Kevin Lam <ke...@shopify.com> 写道：
> 
> Hi all,
> 
> I noticed there isn't much in the way of testing discussed in the Python API docs <https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/python/> for Flink.
> 
> Does the community have any best-practices or recommendations on how testing should be done with PyFlink? 
> 
> Thanks!