You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by narasimha <sw...@gmail.com> on 2020/08/19 10:43:46 UTC

UT/IT Table API/SQL streaming job

Hi,

I'm checking on how to do UT/IT of streaming job written using Table
API/SQL.

I found
https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this
to be useful.

Are there any other recommended libs/ways to do this.

TIA
-- 
A.Narasimha Swamy

Re: UT/IT Table API/SQL streaming job

Posted by narasimha <sw...@gmail.com>.

Thanks Timo,

I'll check it out.

On Wed, Aug 19, 2020 at 5:14 PM Timo Walther <tw...@apache.org> wrote:

> Hi,
>
> this might be helpful as well:
>
>
> https://lists.apache.org/thread.html/rfe3b45a10fc58cf19d2f71c6841515eb7175ba731d5055b06f236b3f%40%3Cuser.flink.apache.org%3E
>
> First of all, it is important to know if you are interested in
> end-to-end tests (incl. connectors) or excluding connectors. If you just
> like to test you operators, you can use a lot of the testing
> infrastructure of Flink.
>
> If your are NOT using event-time, you can simply use
> `TableEnvironment.fromValues()` and `Table.execute().collect()`. This
> test uses it for example [1] (it is one of the newer test generations).
>
> Otherwise you can use or implement your own testing connectors, like in
> org.apache.flink.table.planner.runtime.stream.sql.FunctionITCase#testStructuredScalarFunction
>
> [2].
>
> I hope this helps.
>
> Regards,
> Timo
>
> [1]
>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/MathFunctionsITCase.java
>
> [2]
>
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/FunctionITCase.java#L700
>
> On 19.08.20 12:43, narasimha wrote:
> > Hi,
> >
> > I'm checking on how to do UT/IT of streaming job written using Table
> > API/SQL.
> >
> > I found
> >
> https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this
>
> > to be useful.
> >
> > Are there any other recommended libs/ways to do this.
> >
> > TIA
> > --
> > A.Narasimha Swamy
>
>

-- 
A.Narasimha Swamy

Re: UT/IT Table API/SQL streaming job

Posted by Timo Walther <tw...@apache.org>.

Hi,

this might be helpful as well:

https://lists.apache.org/thread.html/rfe3b45a10fc58cf19d2f71c6841515eb7175ba731d5055b06f236b3f%40%3Cuser.flink.apache.org%3E

First of all, it is important to know if you are interested in 
end-to-end tests (incl. connectors) or excluding connectors. If you just 
like to test you operators, you can use a lot of the testing 
infrastructure of Flink.

If your are NOT using event-time, you can simply use 
`TableEnvironment.fromValues()` and `Table.execute().collect()`. This 
test uses it for example [1] (it is one of the newer test generations).

Otherwise you can use or implement your own testing connectors, like in 
org.apache.flink.table.planner.runtime.stream.sql.FunctionITCase#testStructuredScalarFunction 
[2].

I hope this helps.

Regards,
Timo

[1] 
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/expressions/MathFunctionsITCase.java

[2] 
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/FunctionITCase.java#L700

On 19.08.20 12:43, narasimha wrote:
> Hi,
> 
> I'm checking on how to do UT/IT of streaming job written using Table 
> API/SQL.
> 
> I found 
> https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this 
> to be useful.
> 
> Are there any other recommended libs/ways to do this.
> 
> TIA
> -- 
> A.Narasimha Swamy

Re: UT/IT Table API/SQL streaming job

Posted by narasimha <sw...@gmail.com>.

Hi Till,

Yes, I have gone through the Flink testing documentation.

In Table API/SQL connectors can be abstracted in the query itself, trying
to understand how such pipelines can be tested.
Looking for resources around it.

On Wed, Aug 19, 2020 at 5:15 PM Till Rohrmann <tr...@apache.org> wrote:

> Hi Narasimha,
>
> have you taken a look at the testing documentation [1]? This should give
> you some pointers on how to test Flink jobs.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html
>
> Cheers,
> Till
>
> On Wed, Aug 19, 2020 at 12:44 PM narasimha <sw...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm checking on how to do UT/IT of streaming job written using Table
>> API/SQL.
>>
>> I found
>> https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this
>> to be useful.
>>
>> Are there any other recommended libs/ways to do this.
>>
>> TIA
>> --
>> A.Narasimha Swamy
>>
>

-- 
A.Narasimha Swamy

Re: UT/IT Table API/SQL streaming job

Posted by Till Rohrmann <tr...@apache.org>.

Hi Narasimha,

have you taken a look at the testing documentation [1]? This should give
you some pointers on how to test Flink jobs.

[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/testing.html

Cheers,
Till

On Wed, Aug 19, 2020 at 12:44 PM narasimha <sw...@gmail.com> wrote:

> Hi,
>
> I'm checking on how to do UT/IT of streaming job written using Table
> API/SQL.
>
> I found
> https://stackoverflow.com/questions/54900843/add-a-unit-test-for-flink-sql this
> to be useful.
>
> Are there any other recommended libs/ways to do this.
>
> TIA
> --
> A.Narasimha Swamy
>