Approach to test custom Source/Sink

[Xinbin Huang <bi...@apache.org>]

Hi team,

I'm currently implementing a custom source and sink, and I'm trying to find
a way to test these implementations. The testing section
<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/testing/#unit-testing-stateful-or-timely-udfs--custom-operators>
in the official doc seems to only include testing for stateful/stateless
operators. Do you have any recommendations on how I should approach this?

Thanks
Bin

Re: Approach to test custom Source/Sink

[Caizhi Weng <ts...@gmail.com>]

Hi!

Currently there is no general principle for testing sources and sinks.
However you might want to check out the unit tests and IT cases for Flink
connectors. For example flink-connector-jdbc module has a lot of tests for
the JDBC source and sink. Follow the ideas in these tests should be enough.

Xinbin Huang <bi...@apache.org> 于2021年8月10日周二 上午4:22写道：

> Hi team,
>
> I'm currently implementing a custom source and sink, and I'm trying to
> find a way to test these implementations. The testing section
> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/testing/#unit-testing-stateful-or-timely-udfs--custom-operators>
> in the official doc seems to only include testing for stateful/stateless
> operators. Do you have any recommendations on how I should approach this?
>
> Thanks
> Bin
>

Re: Approach to test custom Source/Sink

[JING ZHANG <be...@gmail.com>]

Hi Bin,
We could try the following method to cover the source/sink test.
Unit test: To verify whether the behavior of each method in custom source
or sink is expected. You could mock interactions with external storage
(database, IO, etc.) in this part.
Integration test: To test whether the source/sink could work well in a real
Flink job. You need to verify whether the custom connector behavior could
well work in a big picture, such as checkpoint, read/ write to external
storage.
You could find many UT/IT in the source code of Flink built-in
connectors[1] .

[1] https://github.com/apache/flink/tree/master/flink-connectors

Best,
JING ZHANG

Xinbin Huang <bi...@apache.org> 于2021年8月10日周二 上午4:22写道：

> Hi team,
>
> I'm currently implementing a custom source and sink, and I'm trying to
> find a way to test these implementations. The testing section
> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/testing/#unit-testing-stateful-or-timely-udfs--custom-operators>
> in the official doc seems to only include testing for stateful/stateless
> operators. Do you have any recommendations on how I should approach this?
>
> Thanks
> Bin
>