[Flink SQL] Run SQL programatically

["cristi.cioriia" <ci...@adobe.com>]

Hey guys,

We have been trying out Fink SQL to implement use-cases like "compute the
last X minutes of data" and we'd like to return the computed data
programatically from Scala/Java, is there a way to write a program to run
SQL queries over Flink in a reactive manner?

We used for now for experimenting purposes the sql-client utility and out
approach was the following:

1. Create the Source table, backed by Kafka.
2. Create the Sink table, also backed by Kafka.
3. Define and run the INSERT query into the sink. We used here hopping
windows, count and group by operators.
4. Define and run the SELECT query to display the data. Here the query uses
sub-selects, to retrieve only results for the last window.

What we would need now is:

1. Define the DDL and DML queries in steps 1-3 into a file and use the
sql-client to send this to the flink cluster for running.What I got from
https://issues.apache.org/jira/browse/FLINK-12828 is that the sql-client
cannot be used for now for this purpose. What would be the alternative here?

2. Write a Scala/Java program that would run the query in step 4 and return
the result to its users, e.g. via a REST interface in a continuous manner,
e.g. by exposing a reactive interface. The JDBC client exposed  by Flink is
not reactive, as the JDBC spec is not reactive. Is there an alternative API
we could use?

3. We would like to get incomplete results for the current minute, e.g. if
we run the query at time 14:34:23, we should get the data for the window
14:30:00 - 14:35:00, not for the window 14:29:00 - 14:34:00.

What would be the best way to achieve these for now? Looking forward for you
answers.

Thanks,
Cristi



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/