[DISCUSS] FLIP-24 - SQL Client

[Timo Walther <tw...@apache.org>]

Hey everyone,

in the past, the community already asked about having a way to write 
Flink jobs without extensive programming skills. During the last year we 
have put a lot of effort in the core of our Table & SQL API. Now it is 
time to improve the tooling around it as well and make Flink more 
accessible. I already opened an issue for adding a SQL CLI client [0]. 
We developed a small protoype at data Artisans that was shown at the 
last Flink Forward Berlin [1].

For Flink 1.5 it would be great to offer at least a CLI client to play 
around with Flink and use it for debugging/demo purposes. We created a 
FLIP-24 [2] that roughly sketches the functionality and architecture. We 
also show how this architecture can evolve from a CLI client to a SQL 
gateway/REST server. Most of it is still up for dicussion. The targeted 
minimum viable product for Flink 1.5 should act as a start for 
collecting feedback and attracting contributors.

So feedback on this FLIP is very welcome. What do you think?

Regards,
Timo


[0] https://issues.apache.org/jira/browse/FLINK-7594
[1] 
https://berlin-2017.flink-forward.org/kb_sessions/from-streams-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client

Re: [DISCUSS] FLIP-24 - SQL Client

[Timo Walther <tw...@apache.org>]

Hi Fabian,

thank you for your feedback. I extended the "Implementation Plan" by 3 
subsections. Regarding the proposed changes, I looked into the code of 
the ClusterClient and talked to Till Rohrmann about it. By now, I could 
not identify features that are missing on existing components. It might 
be necessary to make some internal methods accessible but no big 
changes. I also extended the "Result Retrieval" section to (hopefully) 
answer your questions.

Feel free to give further comments.

Regards,
Timo



Am 12/21/17 um 3:52 PM schrieb Fabian Hueske:
> Hi Timo,
>
> thanks for putting this FLIP together. I think this will be a great feature
> for Flink.
>
> I think it makes sense to show the long term goals of the SQL client/server
> architecture, but the current description makes it a bit difficult to
> figure out what will be part of this FLIP and what will be added later
> because the information is spread across the FLIP. This could be improved
> by describing the "Implementation Plan" in more detail, i.e, have
> subsections for the first version with a detailed list of supported
> features and another for later extensions.
>
> Comments to specific sections:
> - Proposed changes: Are there any changes necessary on existing code (e.g.,
> job submission with multiple jars) or are all changes just additions to
> Flink?
> - Result Retrieval: Extend the table to three columns (Query type, Internal
> mode, External mode).
> - The context of the description of the query types "CREATE MATERIALIZED
> VIEW" and "SELECT" is not clear to me. Add a subsection and explain what is
> described there?
> - Implementation plan: Add which result retrieval modes will be supported
> in the initial version? Which configuration will be available?
>
> Best, Fabian
>
> 2017-12-19 16:53 GMT+01:00 Timo Walther <tw...@apache.org>:
>
>> Hey everyone,
>>
>> in the past, the community already asked about having a way to write Flink
>> jobs without extensive programming skills. During the last year we have put
>> a lot of effort in the core of our Table & SQL API. Now it is time to
>> improve the tooling around it as well and make Flink more accessible. I
>> already opened an issue for adding a SQL CLI client [0]. We developed a
>> small protoype at data Artisans that was shown at the last Flink Forward
>> Berlin [1].
>>
>> For Flink 1.5 it would be great to offer at least a CLI client to play
>> around with Flink and use it for debugging/demo purposes. We created a
>> FLIP-24 [2] that roughly sketches the functionality and architecture. We
>> also show how this architecture can evolve from a CLI client to a SQL
>> gateway/REST server. Most of it is still up for dicussion. The targeted
>> minimum viable product for Flink 1.5 should act as a start for collecting
>> feedback and attracting contributors.
>>
>> So feedback on this FLIP is very welcome. What do you think?
>>
>> Regards,
>> Timo
>>
>>
>> [0] https://issues.apache.org/jira/browse/FLINK-7594
>> [1] https://berlin-2017.flink-forward.org/kb_sessions/from-strea
>> ms-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/
>> [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
>>
>>

Re: [DISCUSS] FLIP-24 - SQL Client

[Fabian Hueske <fh...@gmail.com>]

Hi Timo,

thanks for putting this FLIP together. I think this will be a great feature
for Flink.

I think it makes sense to show the long term goals of the SQL client/server
architecture, but the current description makes it a bit difficult to
figure out what will be part of this FLIP and what will be added later
because the information is spread across the FLIP. This could be improved
by describing the "Implementation Plan" in more detail, i.e, have
subsections for the first version with a detailed list of supported
features and another for later extensions.

Comments to specific sections:
- Proposed changes: Are there any changes necessary on existing code (e.g.,
job submission with multiple jars) or are all changes just additions to
Flink?
- Result Retrieval: Extend the table to three columns (Query type, Internal
mode, External mode).
- The context of the description of the query types "CREATE MATERIALIZED
VIEW" and "SELECT" is not clear to me. Add a subsection and explain what is
described there?
- Implementation plan: Add which result retrieval modes will be supported
in the initial version? Which configuration will be available?

Best, Fabian

2017-12-19 16:53 GMT+01:00 Timo Walther <tw...@apache.org>:

> Hey everyone,
>
> in the past, the community already asked about having a way to write Flink
> jobs without extensive programming skills. During the last year we have put
> a lot of effort in the core of our Table & SQL API. Now it is time to
> improve the tooling around it as well and make Flink more accessible. I
> already opened an issue for adding a SQL CLI client [0]. We developed a
> small protoype at data Artisans that was shown at the last Flink Forward
> Berlin [1].
>
> For Flink 1.5 it would be great to offer at least a CLI client to play
> around with Flink and use it for debugging/demo purposes. We created a
> FLIP-24 [2] that roughly sketches the functionality and architecture. We
> also show how this architecture can evolve from a CLI client to a SQL
> gateway/REST server. Most of it is still up for dicussion. The targeted
> minimum viable product for Flink 1.5 should act as a start for collecting
> feedback and attracting contributors.
>
> So feedback on this FLIP is very welcome. What do you think?
>
> Regards,
> Timo
>
>
> [0] https://issues.apache.org/jira/browse/FLINK-7594
> [1] https://berlin-2017.flink-forward.org/kb_sessions/from-strea
> ms-to-tables-and-back-again-a-demo-of-flinks-table-sql-api/
> [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-24+-+SQL+Client
>
>