Useful methods getting deprecated

[lalala <La...@activist.com>]

Hi Flink users,

I am a big fan of table API and we are extensively using it on petabytes
with ad-hoc queries. In our system, nobody goes and writes table creation
DDL, we inherit table schema from schema registry(avro data in Kafka)
dynamically and create temporary tables in the session. 

I am using methods like `connect`, `registerTableSource` to generate my
tables dynamically which are now giving deprecation warnings. It is just
simply not practical to write SQL for table generation with schema. I may
end-up doing a lot of string operation for generating SQL statement from
Avro Schema for generating tables and use `executeSql`, but I don't think it
will be as cleaner as using `connect`, `registerTableSource`. I am little
bit worried about fetching new Flink releases to my application now. 

Is there any specific reason low level support for creating tables are
getting deprecated?

Best regards,



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Useful methods getting deprecated

[Timo Walther <tw...@apache.org>]

Hi,

I agree that both `connect` and `registerTableSource` are useful for 
generating Table API pipelines. It is likely that both API methods will 
get a replacement in the near future.

Let me explain the current status briefly:

connect(): The CREATE TABLE DDL evolved faster than connect(). The 
method still uses the old pre FLIP-95 interface stack. In order to avoid 
confusion we decided to deprecate it already (without dropping it for 
now). There are plans to reintroduce it

https://cwiki.apache.org/confluence/display/FLINK/FLIP-129%3A+Refactor+Descriptor+API+to+register+connectors+in+Table+API

some prerequisite work has been done in

https://cwiki.apache.org/confluence/display/FLINK/FLIP-164%3A+Improve+Schema+Handling+in+Catalogs


registerTableSource(): This is a bit tricky because the new table 
sources and sinks will work on internal data structures.

The easiest solution is to use the Data Stream API and the new 
`fromDataStream` or `fromChangelogStream` introduced in Flink 1.13.

Otherwise you can implement a helper class that extends 
`ScanTableSource` and uses a `DataStreamScanProvider` with a factory to 
call it via SQL DDL. We might provide such a wrapper class in the near 
future for use cases like yours.

I hope this helps. Feel free to use the deprecated method longer until 
the community provides alternatives that are more helpful.

Thanks for the feedback,
Timo

On 15.05.21 04:11, lalala wrote:
> Hi Flink users,
> 
> I am a big fan of table API and we are extensively using it on petabytes
> with ad-hoc queries. In our system, nobody goes and writes table creation
> DDL, we inherit table schema from schema registry(avro data in Kafka)
> dynamically and create temporary tables in the session.
> 
> I am using methods like `connect`, `registerTableSource` to generate my
> tables dynamically which are now giving deprecation warnings. It is just
> simply not practical to write SQL for table generation with schema. I may
> end-up doing a lot of string operation for generating SQL statement from
> Avro Schema for generating tables and use `executeSql`, but I don't think it
> will be as cleaner as using `connect`, `registerTableSource`. I am little
> bit worried about fetching new Flink releases to my application now.
> 
> Is there any specific reason low level support for creating tables are
> getting deprecated?
> 
> Best regards,
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>