You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Indraneel R <va...@gmail.com> on 2019/09/18 12:36:36 UTC

Property based testing

Hi All,

Is there any property based testing framework for flink like
'SparkTestingBase'  for spark?

Would also be useful to know what are some of the standard testing
practices for data testing for flink pipelines.

regards
-Indraneel

Re: Property based testing

Posted by Indraneel R <va...@gmail.com>.

Oh great! Thanks, Aaron that was quite clear.
I will give it a try!

On Wed, Sep 18, 2019 at 8:29 PM Aaron Levin <aa...@stripe.com> wrote:

> Hey,
>
> I've used ScalaCheck to test flink applications. Basic idea is:
>
> * use ScalaCheck to generate some kind of collection
> * use `fromCollection` in `StreamExecutionEnvironment` to create a
> `DataStream`
> * use `DataStreamUtils.collect` as a sink
> * plug my flink logic between the collection source and the collection sink
> * make a ScalaCheck property assertion based on the input collection and
> output collection.
>
> Possible to wrap all that in a single method in Scala. LMK if you have any
> more questions or any of this was not clear!
>
> (note: not sure how to do this in Java).
>
> Best,
>
> Aaron Levin
>
> On Wed, Sep 18, 2019 at 8:36 AM Indraneel R <va...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Is there any property based testing framework for flink like
>> 'SparkTestingBase'  for spark?
>>
>> Would also be useful to know what are some of the standard testing
>> practices for data testing for flink pipelines.
>>
>> regards
>> -Indraneel
>>
>

Re: Property based testing

Posted by Aaron Levin <aa...@stripe.com>.

Hey,

I've used ScalaCheck to test flink applications. Basic idea is:

* use ScalaCheck to generate some kind of collection
* use `fromCollection` in `StreamExecutionEnvironment` to create a
`DataStream`
* use `DataStreamUtils.collect` as a sink
* plug my flink logic between the collection source and the collection sink
* make a ScalaCheck property assertion based on the input collection and
output collection.

Possible to wrap all that in a single method in Scala. LMK if you have any
more questions or any of this was not clear!

(note: not sure how to do this in Java).

Best,

Aaron Levin

On Wed, Sep 18, 2019 at 8:36 AM Indraneel R <va...@gmail.com> wrote:

> Hi All,
>
> Is there any property based testing framework for flink like
> 'SparkTestingBase'  for spark?
>
> Would also be useful to know what are some of the standard testing
> practices for data testing for flink pipelines.
>
> regards
> -Indraneel
>