[DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi, Flink developers.

I want to start a discussion about the FLIP-91: Support Flink SQL
Gateway[1]. Flink SQL Gateway is a service that allows users to submit and
manage their jobs in the online environment with the pluggable endpoints.
The reason why we introduce the Gateway with pluggable endpoints is that
many users have their preferences. For example, the HiveServer2 users
prefer to use the gateway with HiveServer2-style API, which has numerous
tools. However, some filnk-native users may prefer to use the REST API.
Therefore, we propose the SQL Gateway with pluggable endpoint.

In the FLIP, we also propose the REST endpoint, which has the similar
APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At the
last, we discuss how to use the SQL Client to submit the statement to the
Gateway with the REST API.

I am glad that you can give some feedback about FLIP-91.

Best,
Shengkai

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
[2] https://github.com/ververica/flink-sql-gateway

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi, Paul.

Thanks for you feedback!

> Is operation status exposed to users or just for internal usage to
represent the job status

Only in the synchronized job submission, the operation status contains the
job status. When user chooses the synchronized job submission, it means
- user expects to wait the job finish.
- When users cancels the Operation, it should also cancel the execution of
the job.

Therefore, I think the Operation should also manage the job lifecycle in
the synchronized job submission, which means the JobStatus should be
reflected in the OperationStatus. In the async mode, the Operation comes to
the end when the GatewayService submits the jobs. In this case, we may also
introduce the interface to allow users to get the JobStatus.

> If the cluster is temporarily unavailable, e.g. jobmanager failover, what
status would be
    the operation?

Yes. I will modify the FLIP with your suggestions.

> Does the endpoint pull in the whole result set before returning it to the
client?

No. The endpoint doesn't cache anything. It just deliver the request to the
GatewayService and the GatewayService return the results in the fix-size
buffer.

Best,
Shengkai


Paul Lam <pa...@gmail.com> 于2022年4月21日周四 17:04写道：

> Hi Shengkai,
>
> Good to see FLIP-91 is revisited after such a long time. Big +1 for the
> proposal.
>
> I’ve been using the SQL gateway for a while, just put in my 2 cents:
>
> 1. Is operation status exposed to users or just for internal usage to
> represent the job status?
>     I’m assuming the latter, or else it might confuse users with job
> status and operation status.
>
> 2. If the cluster is temporarily unavailable, e.g. jobmanager failover,
> what status would be
>     the operation? I think PENDING should be better than ERROR.
>
> 3. WRT synchronized job submission, does the endpoint pull in the whole
> result set before
>     returning it to the client?
>
> Thanks a lot!
>
> Best,
> Paul Lam
>
> > 2022年4月21日 14:42，Shengkai Fang <fs...@gmail.com> 写道：
> >
> > Hi, Flink developers.
> >
> > I want to start a discussion about the FLIP-91: Support Flink SQL
> > Gateway[1]. Flink SQL Gateway is a service that allows users to submit
> and
> > manage their jobs in the online environment with the pluggable endpoints.
> > The reason why we introduce the Gateway with pluggable endpoints is that
> > many users have their preferences. For example, the HiveServer2 users
> > prefer to use the gateway with HiveServer2-style API, which has numerous
> > tools. However, some filnk-native users may prefer to use the REST API.
> > Therefore, we propose the SQL Gateway with pluggable endpoint.
> >
> > In the FLIP, we also propose the REST endpoint, which has the similar
> > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At
> the
> > last, we discuss how to use the SQL Client to submit the statement to the
> > Gateway with the REST API.
> >
> > I am glad that you can give some feedback about FLIP-91.
> >
> > Best,
> > Shengkai
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > [2] https://github.com/ververica/flink-sql-gateway
>
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Paul Lam <pa...@gmail.com>]

Hi Shengkai,

Good to see FLIP-91 is revisited after such a long time. Big +1 for the proposal.

I’ve been using the SQL gateway for a while, just put in my 2 cents:

1. Is operation status exposed to users or just for internal usage to represent the job status? 
    I’m assuming the latter, or else it might confuse users with job status and operation status.

2. If the cluster is temporarily unavailable, e.g. jobmanager failover, what status would be 
    the operation? I think PENDING should be better than ERROR.

3. WRT synchronized job submission, does the endpoint pull in the whole result set before 
    returning it to the client? 

Thanks a lot!

Best,
Paul Lam

> 2022年4月21日 14:42，Shengkai Fang <fs...@gmail.com> 写道：
> 
> Hi, Flink developers.
> 
> I want to start a discussion about the FLIP-91: Support Flink SQL
> Gateway[1]. Flink SQL Gateway is a service that allows users to submit and
> manage their jobs in the online environment with the pluggable endpoints.
> The reason why we introduce the Gateway with pluggable endpoints is that
> many users have their preferences. For example, the HiveServer2 users
> prefer to use the gateway with HiveServer2-style API, which has numerous
> tools. However, some filnk-native users may prefer to use the REST API.
> Therefore, we propose the SQL Gateway with pluggable endpoint.
> 
> In the FLIP, we also propose the REST endpoint, which has the similar
> APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At the
> last, we discuss how to use the SQL Client to submit the statement to the
> Gateway with the REST API.
> 
> I am glad that you can give some feedback about FLIP-91.
> 
> Best,
> Shengkai
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> [2] https://github.com/ververica/flink-sql-gateway

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jark Wu <im...@gmail.com>]

Thank Shengkai for the new round of updating.
I don't have comments on the updates.
+1 for starting the vote.

Best,
Jark

On Thu, 19 May 2022 at 20:18, Shengkai Fang <fs...@gmail.com> wrote:

> Hi, Timo.
>
> Thanks for your feedback!
>
>  > SQLGatewayService.getFunction / UserDefinedFunctionInfo
>
> Yes. I miss some parts in the FLIP. I have fix the errors now.
>
>  > configure_session
>
> Thanks for your inputs. Considering the difference, I am still prone to use
> the `configure_session`.
>
>  > `./sql-gateway.sh`
>
> Yes, you are right. We should add a new startup script in the distribution.
> I update the FLIP and add the description to uses the script. Users can
> - start the server with `./sql-gateway.sh start -Dkey=value`
> - stop the last started server  with `./sql-gateway.sh stop`
> - stop all the running server with `./sql-gateway.sh stop-all`
>
> >  an "-e" converts a client to a server
> The "-e" options is just used to connect to the Gateway with the specified
> endpoint. It doesn't  convert the client to the server.
>
> Thank everyone for all the inputs and discussion. If no other problems, I
> think we can restart the voting tomorrow.
>
> Best,
> Shengkai
>
>
> Timo Walther <tw...@apache.org> 于2022年5月19日周四 15:27写道：
>
> > Hi Shengkai,
> >
> > thanks again for the update. I don't have much to add:
> >
> >  > I still think we should use a new state machine
> >
> > Thanks for the explanation about async/sync behavior. I thought we also
> > want to use the Gateway for job status updates. But if FINISHED only
> > refers to the job submission, the new state machine makes more sense.
> > Nevertheless, checking the job status will be a frequent request esp.
> > also for the new lifecycle statements. But I agree to the current design
> > that this is different to state of the operation.
> >
> >  > SQLGatewayService.getFunction / UserDefinedFunctionInfo
> >
> > You forgot to update the class. There is still a UserDefinedFunctionInfo
> > in the FLIP.
> >
> >  > configure_session
> >
> > I don't have a strong opinion on the naming of this particular REST
> > path. But my concern is that we should introduce a term of this family
> > of init/configure statements. Because in SQL Client we call it `init
> > statements` and in the gateway we call it `configuration statements`,
> > but in the end is all statements that are `not DML and DQL`.
> >
> >  > './sql-client.h -e'
> >
> > Would it make sense to introduce a separate `./sql-gateway.sh`? I find
> > it a bit confusing that an "-e" converts a client to a server. Under the
> > hood we can still share the same classes but make it a bit more explicit
> > in the distribution (also for marketing purposes of this feature).
> >
> > Please continue with the voting afterwards.
> >
> > Regards,
> > Timo
> >
> > On 17.05.22 09:14, Shengkai Fang wrote:
> > > Hi, all.
> > >
> > > After discussing with Becket Qin offline, I modified the FLIP a little.
> > The
> > > change is as follow:
> > >
> > > 1. Add /api_versions in the REST API.
> > >
> > > The api is used by the client to know the current REST Endpoint
> supports
> > > which version. The client is able to choose the version for later
> > > communication.
> > >
> > > 2. SqlClient uses -e option to input the endpoint address and port.
> > >
> > > Because the -h option has already been used by the SqlClient. We use
> the
> > > -e, --endpoint for SqlClient to input the address:port of the endpoint.
> > >
> > > 3. Use './sql-client.h -e' to start the gateway mode rather than
> > > '/sql-client.h gateway -e'.
> > >
> > > If the user specifies the -e option, it definitely means to use the
> > gateway
> > > mode. Therefore, it is redundant to use another keyword to indicate the
> > > mode.
> > >
> > > Best,
> > > Shengkai
> > >
> > > Shengkai Fang <fs...@gmail.com> 于2022年5月17日周二 14:13写道：
> > >
> > >> Hi, Jark, Timo. Nice to have an agreement!
> > >>
> > >> Thanks for Jark's inputs about the multiple version Flink. I have
> > already
> > >> updated the FLIP in the rejected alternatives about details.
> > >>
> > >> 1. We should definitely just use LogicalTypeJsonSerializer and not a
> > >> second JSON representation.
> > >>
> > >> Our concern is mainly that it's hard for users to use because of the
> > >> flexible structure. The LogicalTypeJsonSerializer will serialize the
> > >> VARCHAR to "VARCHAR(<LENGTH>)" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\":
> > 0}",
> > >> which requires the end users to process the different situations. But
> in
> > >> some cases, users just print the json to the terminal/web UI.  WDYT?
> > >>
> > >>> Serialize the RowData
> > >>
> > >> Sure. I will keep your advice in mind. I think the current
> serialization
> > >> of the RowData will not use the column name as the Object key in the
> > json.
> > >> I am not sure whether I missed something. It would be nice if you can
> > give
> > >> me an example if I do something wrong.
> > >>
> > >>> Have you also thought about using Flink's state types from Flink
> > >> tasks/jobs?
> > >>
> > >> Yes. But I still think we should use a new state machine. First of
> all,
> > >> Operation in the FLIP is much different from the Job. Operations
> include
> > >> DDL, DML and so on. So it's not suitable to use the small concept to
> > >> replace the big concept. Actually some status in the JobStatus, e.g.
> > >> RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.
> > >>
> > >> On the other hand, the Gateway allows users to submit jobs(DML) in
> > >> sync/async mode. The running status in the Operation Status in the
> > >> different mode has different meaning:
> > >> - In the async mode, when the gateway submits the job, the state comes
> > to
> > >> the FINISHED state
> > >> - In the sync mode, the running status in the Operation status
> includes
> > >> submitting the job, running job. Even if a failover occurs, we still
> > think
> > >> that this Operation is in the RUNNING state. Unless the job is
> > >> unrecoverable, we change the Operation status to ERROR.
> > >>
> > >> Therefore, I think these two concepts are not consistent and we should
> > not
> > >> reuse the JobStatus. I add a section in the rejected alternatives.
> > >>
> > >>> Options to configure the REST endpoint
> > >>
> > >> Yes. I have modified the FLIP about this.
> > >>
> > >>> Naming conversion
> > >>
> > >> Yes. I have modified the FLIP with your suggestions.
> > >>
> > >>> Another smaller shortcomings in the FLIP
> > >>
> > >>>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
> > >>
> > >> After reviewing the java.sql.DatabaseMetaData#getFunctions's java
> doc, I
> > >> find it will return the system and user functions available in the
> > Catalog.
> > >> I think you are right. Therefore, we'd better to rename to the
> > >> listFunctions(SessionHandle sessionHandle, OperationHandle
> > operationHandle,
> > >> String catalog, String database, ShowFunctionsOperation.FunctionScope)
> > and
> > >> it returns FunctionInfo.
> > >>
> > >>>> SQLGatewayService.getGatewayInfo()/getSessionConfig
> > >>
> > >> The result of the SQLGatewayService.getGatewayInfo and
> getSessionConfig
> > is
> > >> not used by the endpoint. The endpoint just serializes the result and
> > >> presents it to the users. If we use the ReadableConfig, it's hard for
> > us to
> > >> iterate all the key value pairs.
> > >>
> > >>> configure_session VS initialize_session
> > >>>> If calling it initialize_session, should we limit it only being
> called
> > >> once?
> > >>
> > >> If we limit it only being called once, it allows the input of the
> > >> initialize_session script. But the current design in the Gateway is
> > aligned
> > >> with the TableEnvironment#executeSql. That is, the input of the
> > statement
> > >> is a single statement rather than the script. Considering the API in
> the
> > >> FLIP is not as same as the initialization in the CLI, I think we can
> use
> > >> the configure_session? What do you think, Timo?
> > >>
> > >> Best,
> > >> Shengkai
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Timo Walther <tw...@apache.org> 于2022年5月16日周一 14:28写道：
> > >>
> > >>> Hi Shengkai, Hi Jark,
> > >>>
> > >>> thanks for the additional explanation and the update of the FLIP.
> This
> > >>> will help us in the future for documenting our decisions. The
> arguments
> > >>> why to include the Gateway into the main repo make a lot of sense to
> > me.
> > >>> Esp. also because both CLI and gateway need some parsing
> functionality
> > >>> that is dependent on the current state of the SQL syntax.
> > >>>
> > >>> Here is my last set of feedback, other than that +1 for the proposal:
> > >>>
> > >>> Serialize the LogicalType
> > >>>
> > >>> The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is
> > >>> different from the current master. We are using the serializable
> > >>> representation of LogicalType as much as possible nowadays. We should
> > >>> definitely just use LogicalTypeJsonSerializer and not a second JSON
> > >>> representation.
> > >>>
> > >>> 1) Serialize the RowData
> > >>>
> > >>> Side note for serializing ROWs: we should not use field names in JSON
> > >>> object keys. As e.g. `null` and other names with special characters
> > >>> cause issues in JSON.
> > >>>
> > >>> 2) We propose the state machine like HiveServer2
> > >>>
> > >>> Have you also thought about using Flink's state types from Flink
> > >>> tasks/jobs? If we were using Flink types directly, it would be easier
> > to
> > >>> monitor the execution of a INSERT INTO job via the gateway without
> > >>> having to map state types. Monitoring jobs is the most important
> > >>> functionality and should be in sync with regular Flink job
> monitoring.
> > A
> > >>> HiveServer2 endpoint can still perform mapping if necessary, but
> within
> > >>> the Flink code base we use a consistent state transition scheme.
> > >>>
> > >>> 3) Options to configure the REST endpoint
> > >>>
> > >>> Given that we also want to be able to put endpoint options into
> > >>> flink-conf.yaml, we should make those option a bit more globally
> > >>> understandable:
> > >>>
> > >>> endpoint.type -> sql-gateway.endpoint.type
> > >>> endpoint.rest.port -> sql-gateway.endpoint.rest.port
> > >>> ...
> > >>>
> > >>> 4) Naming conversion
> > >>>
> > >>> This is very nit picking, but please make sure to name the interfaces
> > >>> consistently for abbreviations. Currently, we mostly use
> `SqlInterface`
> > >>> instead of `SQLInterface`. So in the FLIP:
> > >>>
> > >>> SQLGatewayEndpoint -> SqlGatewayEndpoint
> > >>> SQLGatewayEndpointFactory -> SqlGatewayEndpointFactory
> > >>> e.g. for REST we do it already: RestEndpointVersion
> > >>>
> > >>> 5) Another smaller shortcomings in the FLIP
> > >>>
> > >>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
> > >>> Was it a conscious decision to not use FunctionIdentifier here? You
> > will
> > >>> not be able to list system functions.
> > >>>
> > >>> SQLGatewayService.getGatewayInfo()/getSessionConfig
> > >>> Why not returning ReadableConfig here?
> > >>>
> > >>> Thanks,
> > >>> Timo
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On 12.05.22 17:02, Jark Wu wrote:
> > >>>> Hi Shengkai,
> > >>>>
> > >>>> One comment on the configure_session VS initialize_session,
> > >>>> I think configure_session is better than initialize_session,
> > >>>> because "initialize" sounds like this method can only be invoked
> once.
> > >>>> But I can see this method is useful to be invoked several times to
> > >>>> configure
> > >>>> the session during the lifecycle of a session. If calling it
> > >>>> initialize_session,
> > >>>> should we limit it only be called once?
> > >>>>
> > >>>> Best,
> > >>>> Jark
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Thu, 12 May 2022 at 22:38, Jark Wu <imjark@gmail.com
> > >>>> <ma...@gmail.com>> wrote:
> > >>>>
> > >>>>      Hi Timo,
> > >>>>
> > >>>>      I agree with Shengkai that we should keep Gateway in the Flink
> > main
> > >>>>      repo.
> > >>>>      Here are my thoughts:
> > >>>>
> > >>>>      1) It is not feasible and maintainable to invoke different
> > versions
> > >>>>      of Flink dependencies
> > >>>>      in one JVM. I'm afraid it would be a nightmare when the code is
> > >>>>      messed up with all the reflections.
> > >>>>
> > >>>>      2) Actually, it's very mature to support submitting
> > multi-version by
> > >>>>      deploying one
> > >>>>      gateway process for one version. Many systems are following
> this
> > >>>>      architecture as Shengkai
> > >>>>      mentioned above, including the Ververica Platform. And I don't
> > see
> > >>>>      any problem with it.
> > >>>>
> > >>>>      3) As Jingsong said, SQL CLI should build on top of Gateway to
> > share
> > >>>>      code and have the ability to
> > >>>>        connect to Gateway. If SQL CLI is still kept in the Flink
> > repo, we
> > >>>>      have to put Gateway there as well.
> > >>>>
> > >>>>      4) Besides, Gateway is indispensable for a SQL engine (think of
> > >>>>      Trino/Presto, Spark, Hive).
> > >>>>      Otherwise, Flink will always be a processing system. With
> Gateway
> > >>>>      inside the Flink repo,
> > >>>>      we can provide an out-of-box experience as a SQL query engine.
> > Users
> > >>>>      can try out the gateway
> > >>>>      for the latest version when a new version is released.
> > >>>>
> > >>>>      5) Last, Gateway inside the Flink repo can ensure the highest
> > degree
> > >>>>      of version compatibility.
> > >>>>
> > >>>>      Best,
> > >>>>      Jark
> > >>>>
> > >>>>
> > >>>>      On Thu, 12 May 2022 at 19:19, Shengkai Fang <fskmine@gmail.com
> > >>>>      <ma...@gmail.com>> wrote:
> > >>>>
> > >>>>          Hi Timo.
> > >>>>
> > >>>>          Thanks for your feedback!
> > >>>>
> > >>>>           > It would be great if you can copy/paste some of the
> tricky
> > >>> design
> > >>>>          decisions into the FLIP.
> > >>>>
> > >>>>          Yes. I have already added a section about the `Rejected
> > >>>>          Alternatives`. It
> > >>>>          includes the discuss about the
> > >>>>          - TableInfo and FunctionInfo VS CatalogTable and
> > CatalogFunction
> > >>>>          - Support the multi-version Flink in the Gateway VS Support
> > the
> > >>>>          multi-version in the external Service
> > >>>>          - Merge Gateway into the Flink code base VS Support Gateway
> > in
> > >>>>          the another
> > >>>>          repo
> > >>>>          ...
> > >>>>
> > >>>>
> > >>>>           > Separate code base
> > >>>>
> > >>>>          Let me summarize all the discussion about the separate code
> > >>>>          base. Please
> > >>>>          correct me if anything is wrong.
> > >>>>
> > >>>>
> > >>>>          About support for the Gateway in the Flink code base.
> > >>>>
> > >>>>
> > >>>>          1. It reduces the cost including tests, release and
> > document. We
> > >>>>          don't need
> > >>>>          to upgrade the Gateway when Flink releases.
> > >>>>          2. The SQL Client has the ability to submit the SQL to the
> > >>>>          Gateway. If we
> > >>>>          move the Gateway to the outside repo, the SQL Client
> doesn't
> > >>>>          have the
> > >>>>          related dependencies.
> > >>>>          3. It is benefit for us to reuse code in the same repo.
> > >>>>
> > >>>>          About supporting the Gateway in another repo.
> > >>>>
> > >>>>          1. It is easier to support the multi-version Flink;
> > >>>>
> > >>>>
> > >>>>          First of all, I think we should support the multiple Flink
> > >>>>          version with
> > >>>>          another component that uses the Gateway from the individual
> > >>>>          Flink releases.
> > >>>>          Zeppelin, Livy, and Kyuubi all use similar architecture.
> > >>>>
> > >>>>          The current Gateway in the FLIP is bound to the specific
> > Flink
> > >>>>          version,
> > >>>>          which needs to compile the SQL and submit the job to the
> > cluster
> > >>>>          with the
> > >>>>          specified Flink version.
> > >>>>          We can introduce a service in the future(Maybe we can name
> it
> > >>>>          MultipleVersionFlinkService? ). It may tell the client
> where
> > the
> > >>>>          Gateway
> > >>>>          with specified version is and the client just communicate
> > with
> > >>>>          the Gateway
> > >>>>          later. It may also just parse the request,  convert the
> > request
> > >>>>          to the REST
> > >>>>          API in the FLIP and send to the Gateway with specified
> > version.
> > >>>>
> > >>>>          Therefore, I think we should merge the SQL Gateway inside
> the
> > >>>>          Flink repo.
> > >>>>          The MultipleVersionFlinkService may be inside or outside
> the
> > >>> repo.
> > >>>>
> > >>>>          I add more details in the `Rejected Alternatives` in the
> > FLIP.
> > >>>>
> > >>>>           > Missing REST API spec
> > >>>>
> > >>>>          Good point! I add a section in the `Appendix` about the
> > >>>>          serialization of
> > >>>>          the ResultSet. It includes how to serialize the schema,
> > RowData
> > >>> and
> > >>>>          exceptions.
> > >>>>
> > >>>>           > Consistency of concepts
> > >>>>
> > >>>>          configure_session VS initialize_session
> > >>>>
> > >>>>          Okay. I think it's better to use the initialize_session.
> > >>>>
> > >>>>          TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and
> > >>>>          CatalogFunction
> > >>>>
> > >>>>          The CatalogTable and CatalogFunction are much heavier than
> > the
> > >>>>          TableInfo
> > >>>>          and UserDefinedFunction. The CatalogManager requires
> reading
> > >>>>          from the
> > >>>>          Catalog to get the schema. But in the listTables only care
> > about
> > >>>>          the table
> > >>>>          name, which is much lighter. Therefore, we propose to use
> the
> > >>>>          TableInfo
> > >>>>          with required fields.
> > >>>>
> > >>>>           > Result Retrieval
> > >>>>
> > >>>>          Yes. Currently the API is only used for the preview. I
> added
> > a
> > >>>>          section in
> > >>>>          the `Rejected Alternatives` about this.
> > >>>>
> > >>>>          Best,
> > >>>>          Shengkai
> > >>>>
> > >>>
> > >>>
> > >
> >
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi, Timo.

Thanks for your feedback!

 > SQLGatewayService.getFunction / UserDefinedFunctionInfo

Yes. I miss some parts in the FLIP. I have fix the errors now.

 > configure_session

Thanks for your inputs. Considering the difference, I am still prone to use
the `configure_session`.

 > `./sql-gateway.sh`

Yes, you are right. We should add a new startup script in the distribution.
I update the FLIP and add the description to uses the script. Users can
- start the server with `./sql-gateway.sh start -Dkey=value`
- stop the last started server  with `./sql-gateway.sh stop`
- stop all the running server with `./sql-gateway.sh stop-all`

>  an "-e" converts a client to a server
The "-e" options is just used to connect to the Gateway with the specified
endpoint. It doesn't  convert the client to the server.

Thank everyone for all the inputs and discussion. If no other problems, I
think we can restart the voting tomorrow.

Best,
Shengkai


Timo Walther <tw...@apache.org> 于2022年5月19日周四 15:27写道：

> Hi Shengkai,
>
> thanks again for the update. I don't have much to add:
>
>  > I still think we should use a new state machine
>
> Thanks for the explanation about async/sync behavior. I thought we also
> want to use the Gateway for job status updates. But if FINISHED only
> refers to the job submission, the new state machine makes more sense.
> Nevertheless, checking the job status will be a frequent request esp.
> also for the new lifecycle statements. But I agree to the current design
> that this is different to state of the operation.
>
>  > SQLGatewayService.getFunction / UserDefinedFunctionInfo
>
> You forgot to update the class. There is still a UserDefinedFunctionInfo
> in the FLIP.
>
>  > configure_session
>
> I don't have a strong opinion on the naming of this particular REST
> path. But my concern is that we should introduce a term of this family
> of init/configure statements. Because in SQL Client we call it `init
> statements` and in the gateway we call it `configuration statements`,
> but in the end is all statements that are `not DML and DQL`.
>
>  > './sql-client.h -e'
>
> Would it make sense to introduce a separate `./sql-gateway.sh`? I find
> it a bit confusing that an "-e" converts a client to a server. Under the
> hood we can still share the same classes but make it a bit more explicit
> in the distribution (also for marketing purposes of this feature).
>
> Please continue with the voting afterwards.
>
> Regards,
> Timo
>
> On 17.05.22 09:14, Shengkai Fang wrote:
> > Hi, all.
> >
> > After discussing with Becket Qin offline, I modified the FLIP a little.
> The
> > change is as follow:
> >
> > 1. Add /api_versions in the REST API.
> >
> > The api is used by the client to know the current REST Endpoint supports
> > which version. The client is able to choose the version for later
> > communication.
> >
> > 2. SqlClient uses -e option to input the endpoint address and port.
> >
> > Because the -h option has already been used by the SqlClient. We use the
> > -e, --endpoint for SqlClient to input the address:port of the endpoint.
> >
> > 3. Use './sql-client.h -e' to start the gateway mode rather than
> > '/sql-client.h gateway -e'.
> >
> > If the user specifies the -e option, it definitely means to use the
> gateway
> > mode. Therefore, it is redundant to use another keyword to indicate the
> > mode.
> >
> > Best,
> > Shengkai
> >
> > Shengkai Fang <fs...@gmail.com> 于2022年5月17日周二 14:13写道：
> >
> >> Hi, Jark, Timo. Nice to have an agreement!
> >>
> >> Thanks for Jark's inputs about the multiple version Flink. I have
> already
> >> updated the FLIP in the rejected alternatives about details.
> >>
> >> 1. We should definitely just use LogicalTypeJsonSerializer and not a
> >> second JSON representation.
> >>
> >> Our concern is mainly that it's hard for users to use because of the
> >> flexible structure. The LogicalTypeJsonSerializer will serialize the
> >> VARCHAR to "VARCHAR(<LENGTH>)" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\":
> 0}",
> >> which requires the end users to process the different situations. But in
> >> some cases, users just print the json to the terminal/web UI.  WDYT?
> >>
> >>> Serialize the RowData
> >>
> >> Sure. I will keep your advice in mind. I think the current serialization
> >> of the RowData will not use the column name as the Object key in the
> json.
> >> I am not sure whether I missed something. It would be nice if you can
> give
> >> me an example if I do something wrong.
> >>
> >>> Have you also thought about using Flink's state types from Flink
> >> tasks/jobs?
> >>
> >> Yes. But I still think we should use a new state machine. First of all,
> >> Operation in the FLIP is much different from the Job. Operations include
> >> DDL, DML and so on. So it's not suitable to use the small concept to
> >> replace the big concept. Actually some status in the JobStatus, e.g.
> >> RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.
> >>
> >> On the other hand, the Gateway allows users to submit jobs(DML) in
> >> sync/async mode. The running status in the Operation Status in the
> >> different mode has different meaning:
> >> - In the async mode, when the gateway submits the job, the state comes
> to
> >> the FINISHED state
> >> - In the sync mode, the running status in the Operation status includes
> >> submitting the job, running job. Even if a failover occurs, we still
> think
> >> that this Operation is in the RUNNING state. Unless the job is
> >> unrecoverable, we change the Operation status to ERROR.
> >>
> >> Therefore, I think these two concepts are not consistent and we should
> not
> >> reuse the JobStatus. I add a section in the rejected alternatives.
> >>
> >>> Options to configure the REST endpoint
> >>
> >> Yes. I have modified the FLIP about this.
> >>
> >>> Naming conversion
> >>
> >> Yes. I have modified the FLIP with your suggestions.
> >>
> >>> Another smaller shortcomings in the FLIP
> >>
> >>>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
> >>
> >> After reviewing the java.sql.DatabaseMetaData#getFunctions's java doc, I
> >> find it will return the system and user functions available in the
> Catalog.
> >> I think you are right. Therefore, we'd better to rename to the
> >> listFunctions(SessionHandle sessionHandle, OperationHandle
> operationHandle,
> >> String catalog, String database, ShowFunctionsOperation.FunctionScope)
> and
> >> it returns FunctionInfo.
> >>
> >>>> SQLGatewayService.getGatewayInfo()/getSessionConfig
> >>
> >> The result of the SQLGatewayService.getGatewayInfo and getSessionConfig
> is
> >> not used by the endpoint. The endpoint just serializes the result and
> >> presents it to the users. If we use the ReadableConfig, it's hard for
> us to
> >> iterate all the key value pairs.
> >>
> >>> configure_session VS initialize_session
> >>>> If calling it initialize_session, should we limit it only being called
> >> once?
> >>
> >> If we limit it only being called once, it allows the input of the
> >> initialize_session script. But the current design in the Gateway is
> aligned
> >> with the TableEnvironment#executeSql. That is, the input of the
> statement
> >> is a single statement rather than the script. Considering the API in the
> >> FLIP is not as same as the initialization in the CLI, I think we can use
> >> the configure_session? What do you think, Timo?
> >>
> >> Best,
> >> Shengkai
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Timo Walther <tw...@apache.org> 于2022年5月16日周一 14:28写道：
> >>
> >>> Hi Shengkai, Hi Jark,
> >>>
> >>> thanks for the additional explanation and the update of the FLIP. This
> >>> will help us in the future for documenting our decisions. The arguments
> >>> why to include the Gateway into the main repo make a lot of sense to
> me.
> >>> Esp. also because both CLI and gateway need some parsing functionality
> >>> that is dependent on the current state of the SQL syntax.
> >>>
> >>> Here is my last set of feedback, other than that +1 for the proposal:
> >>>
> >>> Serialize the LogicalType
> >>>
> >>> The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is
> >>> different from the current master. We are using the serializable
> >>> representation of LogicalType as much as possible nowadays. We should
> >>> definitely just use LogicalTypeJsonSerializer and not a second JSON
> >>> representation.
> >>>
> >>> 1) Serialize the RowData
> >>>
> >>> Side note for serializing ROWs: we should not use field names in JSON
> >>> object keys. As e.g. `null` and other names with special characters
> >>> cause issues in JSON.
> >>>
> >>> 2) We propose the state machine like HiveServer2
> >>>
> >>> Have you also thought about using Flink's state types from Flink
> >>> tasks/jobs? If we were using Flink types directly, it would be easier
> to
> >>> monitor the execution of a INSERT INTO job via the gateway without
> >>> having to map state types. Monitoring jobs is the most important
> >>> functionality and should be in sync with regular Flink job monitoring.
> A
> >>> HiveServer2 endpoint can still perform mapping if necessary, but within
> >>> the Flink code base we use a consistent state transition scheme.
> >>>
> >>> 3) Options to configure the REST endpoint
> >>>
> >>> Given that we also want to be able to put endpoint options into
> >>> flink-conf.yaml, we should make those option a bit more globally
> >>> understandable:
> >>>
> >>> endpoint.type -> sql-gateway.endpoint.type
> >>> endpoint.rest.port -> sql-gateway.endpoint.rest.port
> >>> ...
> >>>
> >>> 4) Naming conversion
> >>>
> >>> This is very nit picking, but please make sure to name the interfaces
> >>> consistently for abbreviations. Currently, we mostly use `SqlInterface`
> >>> instead of `SQLInterface`. So in the FLIP:
> >>>
> >>> SQLGatewayEndpoint -> SqlGatewayEndpoint
> >>> SQLGatewayEndpointFactory -> SqlGatewayEndpointFactory
> >>> e.g. for REST we do it already: RestEndpointVersion
> >>>
> >>> 5) Another smaller shortcomings in the FLIP
> >>>
> >>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
> >>> Was it a conscious decision to not use FunctionIdentifier here? You
> will
> >>> not be able to list system functions.
> >>>
> >>> SQLGatewayService.getGatewayInfo()/getSessionConfig
> >>> Why not returning ReadableConfig here?
> >>>
> >>> Thanks,
> >>> Timo
> >>>
> >>>
> >>>
> >>>
> >>> On 12.05.22 17:02, Jark Wu wrote:
> >>>> Hi Shengkai,
> >>>>
> >>>> One comment on the configure_session VS initialize_session,
> >>>> I think configure_session is better than initialize_session,
> >>>> because "initialize" sounds like this method can only be invoked once.
> >>>> But I can see this method is useful to be invoked several times to
> >>>> configure
> >>>> the session during the lifecycle of a session. If calling it
> >>>> initialize_session,
> >>>> should we limit it only be called once?
> >>>>
> >>>> Best,
> >>>> Jark
> >>>>
> >>>>
> >>>>
> >>>> On Thu, 12 May 2022 at 22:38, Jark Wu <imjark@gmail.com
> >>>> <ma...@gmail.com>> wrote:
> >>>>
> >>>>      Hi Timo,
> >>>>
> >>>>      I agree with Shengkai that we should keep Gateway in the Flink
> main
> >>>>      repo.
> >>>>      Here are my thoughts:
> >>>>
> >>>>      1) It is not feasible and maintainable to invoke different
> versions
> >>>>      of Flink dependencies
> >>>>      in one JVM. I'm afraid it would be a nightmare when the code is
> >>>>      messed up with all the reflections.
> >>>>
> >>>>      2) Actually, it's very mature to support submitting
> multi-version by
> >>>>      deploying one
> >>>>      gateway process for one version. Many systems are following this
> >>>>      architecture as Shengkai
> >>>>      mentioned above, including the Ververica Platform. And I don't
> see
> >>>>      any problem with it.
> >>>>
> >>>>      3) As Jingsong said, SQL CLI should build on top of Gateway to
> share
> >>>>      code and have the ability to
> >>>>        connect to Gateway. If SQL CLI is still kept in the Flink
> repo, we
> >>>>      have to put Gateway there as well.
> >>>>
> >>>>      4) Besides, Gateway is indispensable for a SQL engine (think of
> >>>>      Trino/Presto, Spark, Hive).
> >>>>      Otherwise, Flink will always be a processing system. With Gateway
> >>>>      inside the Flink repo,
> >>>>      we can provide an out-of-box experience as a SQL query engine.
> Users
> >>>>      can try out the gateway
> >>>>      for the latest version when a new version is released.
> >>>>
> >>>>      5) Last, Gateway inside the Flink repo can ensure the highest
> degree
> >>>>      of version compatibility.
> >>>>
> >>>>      Best,
> >>>>      Jark
> >>>>
> >>>>
> >>>>      On Thu, 12 May 2022 at 19:19, Shengkai Fang <fskmine@gmail.com
> >>>>      <ma...@gmail.com>> wrote:
> >>>>
> >>>>          Hi Timo.
> >>>>
> >>>>          Thanks for your feedback!
> >>>>
> >>>>           > It would be great if you can copy/paste some of the tricky
> >>> design
> >>>>          decisions into the FLIP.
> >>>>
> >>>>          Yes. I have already added a section about the `Rejected
> >>>>          Alternatives`. It
> >>>>          includes the discuss about the
> >>>>          - TableInfo and FunctionInfo VS CatalogTable and
> CatalogFunction
> >>>>          - Support the multi-version Flink in the Gateway VS Support
> the
> >>>>          multi-version in the external Service
> >>>>          - Merge Gateway into the Flink code base VS Support Gateway
> in
> >>>>          the another
> >>>>          repo
> >>>>          ...
> >>>>
> >>>>
> >>>>           > Separate code base
> >>>>
> >>>>          Let me summarize all the discussion about the separate code
> >>>>          base. Please
> >>>>          correct me if anything is wrong.
> >>>>
> >>>>
> >>>>          About support for the Gateway in the Flink code base.
> >>>>
> >>>>
> >>>>          1. It reduces the cost including tests, release and
> document. We
> >>>>          don't need
> >>>>          to upgrade the Gateway when Flink releases.
> >>>>          2. The SQL Client has the ability to submit the SQL to the
> >>>>          Gateway. If we
> >>>>          move the Gateway to the outside repo, the SQL Client doesn't
> >>>>          have the
> >>>>          related dependencies.
> >>>>          3. It is benefit for us to reuse code in the same repo.
> >>>>
> >>>>          About supporting the Gateway in another repo.
> >>>>
> >>>>          1. It is easier to support the multi-version Flink;
> >>>>
> >>>>
> >>>>          First of all, I think we should support the multiple Flink
> >>>>          version with
> >>>>          another component that uses the Gateway from the individual
> >>>>          Flink releases.
> >>>>          Zeppelin, Livy, and Kyuubi all use similar architecture.
> >>>>
> >>>>          The current Gateway in the FLIP is bound to the specific
> Flink
> >>>>          version,
> >>>>          which needs to compile the SQL and submit the job to the
> cluster
> >>>>          with the
> >>>>          specified Flink version.
> >>>>          We can introduce a service in the future(Maybe we can name it
> >>>>          MultipleVersionFlinkService? ). It may tell the client where
> the
> >>>>          Gateway
> >>>>          with specified version is and the client just communicate
> with
> >>>>          the Gateway
> >>>>          later. It may also just parse the request,  convert the
> request
> >>>>          to the REST
> >>>>          API in the FLIP and send to the Gateway with specified
> version.
> >>>>
> >>>>          Therefore, I think we should merge the SQL Gateway inside the
> >>>>          Flink repo.
> >>>>          The MultipleVersionFlinkService may be inside or outside the
> >>> repo.
> >>>>
> >>>>          I add more details in the `Rejected Alternatives` in the
> FLIP.
> >>>>
> >>>>           > Missing REST API spec
> >>>>
> >>>>          Good point! I add a section in the `Appendix` about the
> >>>>          serialization of
> >>>>          the ResultSet. It includes how to serialize the schema,
> RowData
> >>> and
> >>>>          exceptions.
> >>>>
> >>>>           > Consistency of concepts
> >>>>
> >>>>          configure_session VS initialize_session
> >>>>
> >>>>          Okay. I think it's better to use the initialize_session.
> >>>>
> >>>>          TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and
> >>>>          CatalogFunction
> >>>>
> >>>>          The CatalogTable and CatalogFunction are much heavier than
> the
> >>>>          TableInfo
> >>>>          and UserDefinedFunction. The CatalogManager requires reading
> >>>>          from the
> >>>>          Catalog to get the schema. But in the listTables only care
> about
> >>>>          the table
> >>>>          name, which is much lighter. Therefore, we propose to use the
> >>>>          TableInfo
> >>>>          with required fields.
> >>>>
> >>>>           > Result Retrieval
> >>>>
> >>>>          Yes. Currently the API is only used for the preview. I added
> a
> >>>>          section in
> >>>>          the `Rejected Alternatives` about this.
> >>>>
> >>>>          Best,
> >>>>          Shengkai
> >>>>
> >>>
> >>>
> >
>
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Timo Walther <tw...@apache.org>]

Hi Shengkai,

thanks again for the update. I don't have much to add:

 > I still think we should use a new state machine

Thanks for the explanation about async/sync behavior. I thought we also 
want to use the Gateway for job status updates. But if FINISHED only 
refers to the job submission, the new state machine makes more sense. 
Nevertheless, checking the job status will be a frequent request esp. 
also for the new lifecycle statements. But I agree to the current design 
that this is different to state of the operation.

 > SQLGatewayService.getFunction / UserDefinedFunctionInfo

You forgot to update the class. There is still a UserDefinedFunctionInfo 
in the FLIP.

 > configure_session

I don't have a strong opinion on the naming of this particular REST 
path. But my concern is that we should introduce a term of this family 
of init/configure statements. Because in SQL Client we call it `init 
statements` and in the gateway we call it `configuration statements`, 
but in the end is all statements that are `not DML and DQL`.

 > './sql-client.h -e'

Would it make sense to introduce a separate `./sql-gateway.sh`? I find 
it a bit confusing that an "-e" converts a client to a server. Under the 
hood we can still share the same classes but make it a bit more explicit 
in the distribution (also for marketing purposes of this feature).

Please continue with the voting afterwards.

Regards,
Timo

On 17.05.22 09:14, Shengkai Fang wrote:
> Hi, all.
> 
> After discussing with Becket Qin offline, I modified the FLIP a little. The
> change is as follow:
> 
> 1. Add /api_versions in the REST API.
> 
> The api is used by the client to know the current REST Endpoint supports
> which version. The client is able to choose the version for later
> communication.
> 
> 2. SqlClient uses -e option to input the endpoint address and port.
> 
> Because the -h option has already been used by the SqlClient. We use the
> -e, --endpoint for SqlClient to input the address:port of the endpoint.
> 
> 3. Use './sql-client.h -e' to start the gateway mode rather than
> '/sql-client.h gateway -e'.
> 
> If the user specifies the -e option, it definitely means to use the gateway
> mode. Therefore, it is redundant to use another keyword to indicate the
> mode.
> 
> Best,
> Shengkai
> 
> Shengkai Fang <fs...@gmail.com> 于2022年5月17日周二 14:13写道：
> 
>> Hi, Jark, Timo. Nice to have an agreement!
>>
>> Thanks for Jark's inputs about the multiple version Flink. I have already
>> updated the FLIP in the rejected alternatives about details.
>>
>> 1. We should definitely just use LogicalTypeJsonSerializer and not a
>> second JSON representation.
>>
>> Our concern is mainly that it's hard for users to use because of the
>> flexible structure. The LogicalTypeJsonSerializer will serialize the
>> VARCHAR to "VARCHAR(<LENGTH>)" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\": 0}",
>> which requires the end users to process the different situations. But in
>> some cases, users just print the json to the terminal/web UI.  WDYT?
>>
>>> Serialize the RowData
>>
>> Sure. I will keep your advice in mind. I think the current serialization
>> of the RowData will not use the column name as the Object key in the json.
>> I am not sure whether I missed something. It would be nice if you can give
>> me an example if I do something wrong.
>>
>>> Have you also thought about using Flink's state types from Flink
>> tasks/jobs?
>>
>> Yes. But I still think we should use a new state machine. First of all,
>> Operation in the FLIP is much different from the Job. Operations include
>> DDL, DML and so on. So it's not suitable to use the small concept to
>> replace the big concept. Actually some status in the JobStatus, e.g.
>> RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.
>>
>> On the other hand, the Gateway allows users to submit jobs(DML) in
>> sync/async mode. The running status in the Operation Status in the
>> different mode has different meaning:
>> - In the async mode, when the gateway submits the job, the state comes to
>> the FINISHED state
>> - In the sync mode, the running status in the Operation status includes
>> submitting the job, running job. Even if a failover occurs, we still think
>> that this Operation is in the RUNNING state. Unless the job is
>> unrecoverable, we change the Operation status to ERROR.
>>
>> Therefore, I think these two concepts are not consistent and we should not
>> reuse the JobStatus. I add a section in the rejected alternatives.
>>
>>> Options to configure the REST endpoint
>>
>> Yes. I have modified the FLIP about this.
>>
>>> Naming conversion
>>
>> Yes. I have modified the FLIP with your suggestions.
>>
>>> Another smaller shortcomings in the FLIP
>>
>>>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
>>
>> After reviewing the java.sql.DatabaseMetaData#getFunctions's java doc, I
>> find it will return the system and user functions available in the Catalog.
>> I think you are right. Therefore, we'd better to rename to the
>> listFunctions(SessionHandle sessionHandle, OperationHandle operationHandle,
>> String catalog, String database, ShowFunctionsOperation.FunctionScope) and
>> it returns FunctionInfo.
>>
>>>> SQLGatewayService.getGatewayInfo()/getSessionConfig
>>
>> The result of the SQLGatewayService.getGatewayInfo and getSessionConfig is
>> not used by the endpoint. The endpoint just serializes the result and
>> presents it to the users. If we use the ReadableConfig, it's hard for us to
>> iterate all the key value pairs.
>>
>>> configure_session VS initialize_session
>>>> If calling it initialize_session, should we limit it only being called
>> once?
>>
>> If we limit it only being called once, it allows the input of the
>> initialize_session script. But the current design in the Gateway is aligned
>> with the TableEnvironment#executeSql. That is, the input of the statement
>> is a single statement rather than the script. Considering the API in the
>> FLIP is not as same as the initialization in the CLI, I think we can use
>> the configure_session? What do you think, Timo?
>>
>> Best,
>> Shengkai
>>
>>
>>
>>
>>
>>
>>
>> Timo Walther <tw...@apache.org> 于2022年5月16日周一 14:28写道：
>>
>>> Hi Shengkai, Hi Jark,
>>>
>>> thanks for the additional explanation and the update of the FLIP. This
>>> will help us in the future for documenting our decisions. The arguments
>>> why to include the Gateway into the main repo make a lot of sense to me.
>>> Esp. also because both CLI and gateway need some parsing functionality
>>> that is dependent on the current state of the SQL syntax.
>>>
>>> Here is my last set of feedback, other than that +1 for the proposal:
>>>
>>> Serialize the LogicalType
>>>
>>> The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is
>>> different from the current master. We are using the serializable
>>> representation of LogicalType as much as possible nowadays. We should
>>> definitely just use LogicalTypeJsonSerializer and not a second JSON
>>> representation.
>>>
>>> 1) Serialize the RowData
>>>
>>> Side note for serializing ROWs: we should not use field names in JSON
>>> object keys. As e.g. `null` and other names with special characters
>>> cause issues in JSON.
>>>
>>> 2) We propose the state machine like HiveServer2
>>>
>>> Have you also thought about using Flink's state types from Flink
>>> tasks/jobs? If we were using Flink types directly, it would be easier to
>>> monitor the execution of a INSERT INTO job via the gateway without
>>> having to map state types. Monitoring jobs is the most important
>>> functionality and should be in sync with regular Flink job monitoring. A
>>> HiveServer2 endpoint can still perform mapping if necessary, but within
>>> the Flink code base we use a consistent state transition scheme.
>>>
>>> 3) Options to configure the REST endpoint
>>>
>>> Given that we also want to be able to put endpoint options into
>>> flink-conf.yaml, we should make those option a bit more globally
>>> understandable:
>>>
>>> endpoint.type -> sql-gateway.endpoint.type
>>> endpoint.rest.port -> sql-gateway.endpoint.rest.port
>>> ...
>>>
>>> 4) Naming conversion
>>>
>>> This is very nit picking, but please make sure to name the interfaces
>>> consistently for abbreviations. Currently, we mostly use `SqlInterface`
>>> instead of `SQLInterface`. So in the FLIP:
>>>
>>> SQLGatewayEndpoint -> SqlGatewayEndpoint
>>> SQLGatewayEndpointFactory -> SqlGatewayEndpointFactory
>>> e.g. for REST we do it already: RestEndpointVersion
>>>
>>> 5) Another smaller shortcomings in the FLIP
>>>
>>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
>>> Was it a conscious decision to not use FunctionIdentifier here? You will
>>> not be able to list system functions.
>>>
>>> SQLGatewayService.getGatewayInfo()/getSessionConfig
>>> Why not returning ReadableConfig here?
>>>
>>> Thanks,
>>> Timo
>>>
>>>
>>>
>>>
>>> On 12.05.22 17:02, Jark Wu wrote:
>>>> Hi Shengkai,
>>>>
>>>> One comment on the configure_session VS initialize_session,
>>>> I think configure_session is better than initialize_session,
>>>> because "initialize" sounds like this method can only be invoked once.
>>>> But I can see this method is useful to be invoked several times to
>>>> configure
>>>> the session during the lifecycle of a session. If calling it
>>>> initialize_session,
>>>> should we limit it only be called once?
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>>
>>>>
>>>> On Thu, 12 May 2022 at 22:38, Jark Wu <imjark@gmail.com
>>>> <ma...@gmail.com>> wrote:
>>>>
>>>>      Hi Timo,
>>>>
>>>>      I agree with Shengkai that we should keep Gateway in the Flink main
>>>>      repo.
>>>>      Here are my thoughts:
>>>>
>>>>      1) It is not feasible and maintainable to invoke different versions
>>>>      of Flink dependencies
>>>>      in one JVM. I'm afraid it would be a nightmare when the code is
>>>>      messed up with all the reflections.
>>>>
>>>>      2) Actually, it's very mature to support submitting multi-version by
>>>>      deploying one
>>>>      gateway process for one version. Many systems are following this
>>>>      architecture as Shengkai
>>>>      mentioned above, including the Ververica Platform. And I don't see
>>>>      any problem with it.
>>>>
>>>>      3) As Jingsong said, SQL CLI should build on top of Gateway to share
>>>>      code and have the ability to
>>>>        connect to Gateway. If SQL CLI is still kept in the Flink repo, we
>>>>      have to put Gateway there as well.
>>>>
>>>>      4) Besides, Gateway is indispensable for a SQL engine (think of
>>>>      Trino/Presto, Spark, Hive).
>>>>      Otherwise, Flink will always be a processing system. With Gateway
>>>>      inside the Flink repo,
>>>>      we can provide an out-of-box experience as a SQL query engine. Users
>>>>      can try out the gateway
>>>>      for the latest version when a new version is released.
>>>>
>>>>      5) Last, Gateway inside the Flink repo can ensure the highest degree
>>>>      of version compatibility.
>>>>
>>>>      Best,
>>>>      Jark
>>>>
>>>>
>>>>      On Thu, 12 May 2022 at 19:19, Shengkai Fang <fskmine@gmail.com
>>>>      <ma...@gmail.com>> wrote:
>>>>
>>>>          Hi Timo.
>>>>
>>>>          Thanks for your feedback!
>>>>
>>>>           > It would be great if you can copy/paste some of the tricky
>>> design
>>>>          decisions into the FLIP.
>>>>
>>>>          Yes. I have already added a section about the `Rejected
>>>>          Alternatives`. It
>>>>          includes the discuss about the
>>>>          - TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
>>>>          - Support the multi-version Flink in the Gateway VS Support the
>>>>          multi-version in the external Service
>>>>          - Merge Gateway into the Flink code base VS Support Gateway in
>>>>          the another
>>>>          repo
>>>>          ...
>>>>
>>>>
>>>>           > Separate code base
>>>>
>>>>          Let me summarize all the discussion about the separate code
>>>>          base. Please
>>>>          correct me if anything is wrong.
>>>>
>>>>
>>>>          About support for the Gateway in the Flink code base.
>>>>
>>>>
>>>>          1. It reduces the cost including tests, release and document. We
>>>>          don't need
>>>>          to upgrade the Gateway when Flink releases.
>>>>          2. The SQL Client has the ability to submit the SQL to the
>>>>          Gateway. If we
>>>>          move the Gateway to the outside repo, the SQL Client doesn't
>>>>          have the
>>>>          related dependencies.
>>>>          3. It is benefit for us to reuse code in the same repo.
>>>>
>>>>          About supporting the Gateway in another repo.
>>>>
>>>>          1. It is easier to support the multi-version Flink;
>>>>
>>>>
>>>>          First of all, I think we should support the multiple Flink
>>>>          version with
>>>>          another component that uses the Gateway from the individual
>>>>          Flink releases.
>>>>          Zeppelin, Livy, and Kyuubi all use similar architecture.
>>>>
>>>>          The current Gateway in the FLIP is bound to the specific Flink
>>>>          version,
>>>>          which needs to compile the SQL and submit the job to the cluster
>>>>          with the
>>>>          specified Flink version.
>>>>          We can introduce a service in the future(Maybe we can name it
>>>>          MultipleVersionFlinkService? ). It may tell the client where the
>>>>          Gateway
>>>>          with specified version is and the client just communicate with
>>>>          the Gateway
>>>>          later. It may also just parse the request,  convert the request
>>>>          to the REST
>>>>          API in the FLIP and send to the Gateway with specified version.
>>>>
>>>>          Therefore, I think we should merge the SQL Gateway inside the
>>>>          Flink repo.
>>>>          The MultipleVersionFlinkService may be inside or outside the
>>> repo.
>>>>
>>>>          I add more details in the `Rejected Alternatives` in the FLIP.
>>>>
>>>>           > Missing REST API spec
>>>>
>>>>          Good point! I add a section in the `Appendix` about the
>>>>          serialization of
>>>>          the ResultSet. It includes how to serialize the schema, RowData
>>> and
>>>>          exceptions.
>>>>
>>>>           > Consistency of concepts
>>>>
>>>>          configure_session VS initialize_session
>>>>
>>>>          Okay. I think it's better to use the initialize_session.
>>>>
>>>>          TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and
>>>>          CatalogFunction
>>>>
>>>>          The CatalogTable and CatalogFunction are much heavier than the
>>>>          TableInfo
>>>>          and UserDefinedFunction. The CatalogManager requires reading
>>>>          from the
>>>>          Catalog to get the schema. But in the listTables only care about
>>>>          the table
>>>>          name, which is much lighter. Therefore, we propose to use the
>>>>          TableInfo
>>>>          with required fields.
>>>>
>>>>           > Result Retrieval
>>>>
>>>>          Yes. Currently the API is only used for the preview. I added a
>>>>          section in
>>>>          the `Rejected Alternatives` about this.
>>>>
>>>>          Best,
>>>>          Shengkai
>>>>
>>>
>>>
> 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi, all.

After discussing with Becket Qin offline, I modified the FLIP a little. The
change is as follow:

1. Add /api_versions in the REST API.

The api is used by the client to know the current REST Endpoint supports
which version. The client is able to choose the version for later
communication.

2. SqlClient uses -e option to input the endpoint address and port.

Because the -h option has already been used by the SqlClient. We use the
-e, --endpoint for SqlClient to input the address:port of the endpoint.

3. Use './sql-client.h -e' to start the gateway mode rather than
'/sql-client.h gateway -e'.

If the user specifies the -e option, it definitely means to use the gateway
mode. Therefore, it is redundant to use another keyword to indicate the
mode.

Best,
Shengkai

Shengkai Fang <fs...@gmail.com> 于2022年5月17日周二 14:13写道：

> Hi, Jark, Timo. Nice to have an agreement!
>
> Thanks for Jark's inputs about the multiple version Flink. I have already
> updated the FLIP in the rejected alternatives about details.
>
> 1. We should definitely just use LogicalTypeJsonSerializer and not a
> second JSON representation.
>
> Our concern is mainly that it's hard for users to use because of the
> flexible structure. The LogicalTypeJsonSerializer will serialize the
> VARCHAR to "VARCHAR(<LENGTH>)" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\": 0}",
> which requires the end users to process the different situations. But in
> some cases, users just print the json to the terminal/web UI.  WDYT?
>
> > Serialize the RowData
>
> Sure. I will keep your advice in mind. I think the current serialization
> of the RowData will not use the column name as the Object key in the json.
> I am not sure whether I missed something. It would be nice if you can give
> me an example if I do something wrong.
>
> > Have you also thought about using Flink's state types from Flink
> tasks/jobs?
>
> Yes. But I still think we should use a new state machine. First of all,
> Operation in the FLIP is much different from the Job. Operations include
> DDL, DML and so on. So it's not suitable to use the small concept to
> replace the big concept. Actually some status in the JobStatus, e.g.
> RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.
>
> On the other hand, the Gateway allows users to submit jobs(DML) in
> sync/async mode. The running status in the Operation Status in the
> different mode has different meaning:
> - In the async mode, when the gateway submits the job, the state comes to
> the FINISHED state
> - In the sync mode, the running status in the Operation status includes
> submitting the job, running job. Even if a failover occurs, we still think
> that this Operation is in the RUNNING state. Unless the job is
> unrecoverable, we change the Operation status to ERROR.
>
> Therefore, I think these two concepts are not consistent and we should not
> reuse the JobStatus. I add a section in the rejected alternatives.
>
> > Options to configure the REST endpoint
>
> Yes. I have modified the FLIP about this.
>
> > Naming conversion
>
> Yes. I have modified the FLIP with your suggestions.
>
> > Another smaller shortcomings in the FLIP
>
> >> SQLGatewayService.getFunction / UserDefinedFunctionInfo
>
> After reviewing the java.sql.DatabaseMetaData#getFunctions's java doc, I
> find it will return the system and user functions available in the Catalog.
> I think you are right. Therefore, we'd better to rename to the
> listFunctions(SessionHandle sessionHandle, OperationHandle operationHandle,
> String catalog, String database, ShowFunctionsOperation.FunctionScope) and
> it returns FunctionInfo.
>
> >> SQLGatewayService.getGatewayInfo()/getSessionConfig
>
> The result of the SQLGatewayService.getGatewayInfo and getSessionConfig is
> not used by the endpoint. The endpoint just serializes the result and
> presents it to the users. If we use the ReadableConfig, it's hard for us to
> iterate all the key value pairs.
>
> > configure_session VS initialize_session
> >> If calling it initialize_session, should we limit it only being called
> once?
>
> If we limit it only being called once, it allows the input of the
> initialize_session script. But the current design in the Gateway is aligned
> with the TableEnvironment#executeSql. That is, the input of the statement
> is a single statement rather than the script. Considering the API in the
> FLIP is not as same as the initialization in the CLI, I think we can use
> the configure_session? What do you think, Timo?
>
> Best,
> Shengkai
>
>
>
>
>
>
>
> Timo Walther <tw...@apache.org> 于2022年5月16日周一 14:28写道：
>
>> Hi Shengkai, Hi Jark,
>>
>> thanks for the additional explanation and the update of the FLIP. This
>> will help us in the future for documenting our decisions. The arguments
>> why to include the Gateway into the main repo make a lot of sense to me.
>> Esp. also because both CLI and gateway need some parsing functionality
>> that is dependent on the current state of the SQL syntax.
>>
>> Here is my last set of feedback, other than that +1 for the proposal:
>>
>> Serialize the LogicalType
>>
>> The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is
>> different from the current master. We are using the serializable
>> representation of LogicalType as much as possible nowadays. We should
>> definitely just use LogicalTypeJsonSerializer and not a second JSON
>> representation.
>>
>> 1) Serialize the RowData
>>
>> Side note for serializing ROWs: we should not use field names in JSON
>> object keys. As e.g. `null` and other names with special characters
>> cause issues in JSON.
>>
>> 2) We propose the state machine like HiveServer2
>>
>> Have you also thought about using Flink's state types from Flink
>> tasks/jobs? If we were using Flink types directly, it would be easier to
>> monitor the execution of a INSERT INTO job via the gateway without
>> having to map state types. Monitoring jobs is the most important
>> functionality and should be in sync with regular Flink job monitoring. A
>> HiveServer2 endpoint can still perform mapping if necessary, but within
>> the Flink code base we use a consistent state transition scheme.
>>
>> 3) Options to configure the REST endpoint
>>
>> Given that we also want to be able to put endpoint options into
>> flink-conf.yaml, we should make those option a bit more globally
>> understandable:
>>
>> endpoint.type -> sql-gateway.endpoint.type
>> endpoint.rest.port -> sql-gateway.endpoint.rest.port
>> ...
>>
>> 4) Naming conversion
>>
>> This is very nit picking, but please make sure to name the interfaces
>> consistently for abbreviations. Currently, we mostly use `SqlInterface`
>> instead of `SQLInterface`. So in the FLIP:
>>
>> SQLGatewayEndpoint -> SqlGatewayEndpoint
>> SQLGatewayEndpointFactory -> SqlGatewayEndpointFactory
>> e.g. for REST we do it already: RestEndpointVersion
>>
>> 5) Another smaller shortcomings in the FLIP
>>
>> SQLGatewayService.getFunction / UserDefinedFunctionInfo
>> Was it a conscious decision to not use FunctionIdentifier here? You will
>> not be able to list system functions.
>>
>> SQLGatewayService.getGatewayInfo()/getSessionConfig
>> Why not returning ReadableConfig here?
>>
>> Thanks,
>> Timo
>>
>>
>>
>>
>> On 12.05.22 17:02, Jark Wu wrote:
>> > Hi Shengkai,
>> >
>> > One comment on the configure_session VS initialize_session,
>> > I think configure_session is better than initialize_session,
>> > because "initialize" sounds like this method can only be invoked once.
>> > But I can see this method is useful to be invoked several times to
>> > configure
>> > the session during the lifecycle of a session. If calling it
>> > initialize_session,
>> > should we limit it only be called once?
>> >
>> > Best,
>> > Jark
>> >
>> >
>> >
>> > On Thu, 12 May 2022 at 22:38, Jark Wu <imjark@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> >     Hi Timo,
>> >
>> >     I agree with Shengkai that we should keep Gateway in the Flink main
>> >     repo.
>> >     Here are my thoughts:
>> >
>> >     1) It is not feasible and maintainable to invoke different versions
>> >     of Flink dependencies
>> >     in one JVM. I'm afraid it would be a nightmare when the code is
>> >     messed up with all the reflections.
>> >
>> >     2) Actually, it's very mature to support submitting multi-version by
>> >     deploying one
>> >     gateway process for one version. Many systems are following this
>> >     architecture as Shengkai
>> >     mentioned above, including the Ververica Platform. And I don't see
>> >     any problem with it.
>> >
>> >     3) As Jingsong said, SQL CLI should build on top of Gateway to share
>> >     code and have the ability to
>> >       connect to Gateway. If SQL CLI is still kept in the Flink repo, we
>> >     have to put Gateway there as well.
>> >
>> >     4) Besides, Gateway is indispensable for a SQL engine (think of
>> >     Trino/Presto, Spark, Hive).
>> >     Otherwise, Flink will always be a processing system. With Gateway
>> >     inside the Flink repo,
>> >     we can provide an out-of-box experience as a SQL query engine. Users
>> >     can try out the gateway
>> >     for the latest version when a new version is released.
>> >
>> >     5) Last, Gateway inside the Flink repo can ensure the highest degree
>> >     of version compatibility.
>> >
>> >     Best,
>> >     Jark
>> >
>> >
>> >     On Thu, 12 May 2022 at 19:19, Shengkai Fang <fskmine@gmail.com
>> >     <ma...@gmail.com>> wrote:
>> >
>> >         Hi Timo.
>> >
>> >         Thanks for your feedback!
>> >
>> >          > It would be great if you can copy/paste some of the tricky
>> design
>> >         decisions into the FLIP.
>> >
>> >         Yes. I have already added a section about the `Rejected
>> >         Alternatives`. It
>> >         includes the discuss about the
>> >         - TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
>> >         - Support the multi-version Flink in the Gateway VS Support the
>> >         multi-version in the external Service
>> >         - Merge Gateway into the Flink code base VS Support Gateway in
>> >         the another
>> >         repo
>> >         ...
>> >
>> >
>> >          > Separate code base
>> >
>> >         Let me summarize all the discussion about the separate code
>> >         base. Please
>> >         correct me if anything is wrong.
>> >
>> >
>> >         About support for the Gateway in the Flink code base.
>> >
>> >
>> >         1. It reduces the cost including tests, release and document. We
>> >         don't need
>> >         to upgrade the Gateway when Flink releases.
>> >         2. The SQL Client has the ability to submit the SQL to the
>> >         Gateway. If we
>> >         move the Gateway to the outside repo, the SQL Client doesn't
>> >         have the
>> >         related dependencies.
>> >         3. It is benefit for us to reuse code in the same repo.
>> >
>> >         About supporting the Gateway in another repo.
>> >
>> >         1. It is easier to support the multi-version Flink;
>> >
>> >
>> >         First of all, I think we should support the multiple Flink
>> >         version with
>> >         another component that uses the Gateway from the individual
>> >         Flink releases.
>> >         Zeppelin, Livy, and Kyuubi all use similar architecture.
>> >
>> >         The current Gateway in the FLIP is bound to the specific Flink
>> >         version,
>> >         which needs to compile the SQL and submit the job to the cluster
>> >         with the
>> >         specified Flink version.
>> >         We can introduce a service in the future(Maybe we can name it
>> >         MultipleVersionFlinkService? ). It may tell the client where the
>> >         Gateway
>> >         with specified version is and the client just communicate with
>> >         the Gateway
>> >         later. It may also just parse the request,  convert the request
>> >         to the REST
>> >         API in the FLIP and send to the Gateway with specified version.
>> >
>> >         Therefore, I think we should merge the SQL Gateway inside the
>> >         Flink repo.
>> >         The MultipleVersionFlinkService may be inside or outside the
>> repo.
>> >
>> >         I add more details in the `Rejected Alternatives` in the FLIP.
>> >
>> >          > Missing REST API spec
>> >
>> >         Good point! I add a section in the `Appendix` about the
>> >         serialization of
>> >         the ResultSet. It includes how to serialize the schema, RowData
>> and
>> >         exceptions.
>> >
>> >          > Consistency of concepts
>> >
>> >         configure_session VS initialize_session
>> >
>> >         Okay. I think it's better to use the initialize_session.
>> >
>> >         TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and
>> >         CatalogFunction
>> >
>> >         The CatalogTable and CatalogFunction are much heavier than the
>> >         TableInfo
>> >         and UserDefinedFunction. The CatalogManager requires reading
>> >         from the
>> >         Catalog to get the schema. But in the listTables only care about
>> >         the table
>> >         name, which is much lighter. Therefore, we propose to use the
>> >         TableInfo
>> >         with required fields.
>> >
>> >          > Result Retrieval
>> >
>> >         Yes. Currently the API is only used for the preview. I added a
>> >         section in
>> >         the `Rejected Alternatives` about this.
>> >
>> >         Best,
>> >         Shengkai
>> >
>>
>>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi, Jark, Timo. Nice to have an agreement!

Thanks for Jark's inputs about the multiple version Flink. I have already
updated the FLIP in the rejected alternatives about details.

1. We should definitely just use LogicalTypeJsonSerializer and not a second
JSON representation.

Our concern is mainly that it's hard for users to use because of the
flexible structure. The LogicalTypeJsonSerializer will serialize the
VARCHAR to "VARCHAR(<LENGTH>)" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\": 0}",
which requires the end users to process the different situations. But in
some cases, users just print the json to the terminal/web UI.  WDYT?

> Serialize the RowData

Sure. I will keep your advice in mind. I think the current serialization of
the RowData will not use the column name as the Object key in the json. I
am not sure whether I missed something. It would be nice if you can give me
an example if I do something wrong.

> Have you also thought about using Flink's state types from Flink
tasks/jobs?

Yes. But I still think we should use a new state machine. First of all,
Operation in the FLIP is much different from the Job. Operations include
DDL, DML and so on. So it's not suitable to use the small concept to
replace the big concept. Actually some status in the JobStatus, e.g.
RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.

On the other hand, the Gateway allows users to submit jobs(DML) in
sync/async mode. The running status in the Operation Status in the
different mode has different meaning:
- In the async mode, when the gateway submits the job, the state comes to
the FINISHED state
- In the sync mode, the running status in the Operation status includes
submitting the job, running job. Even if a failover occurs, we still think
that this Operation is in the RUNNING state. Unless the job is
unrecoverable, we change the Operation status to ERROR.

Therefore, I think these two concepts are not consistent and we should not
reuse the JobStatus. I add a section in the rejected alternatives.

> Options to configure the REST endpoint

Yes. I have modified the FLIP about this.

> Naming conversion

Yes. I have modified the FLIP with your suggestions.

> Another smaller shortcomings in the FLIP

>> SQLGatewayService.getFunction / UserDefinedFunctionInfo

After reviewing the java.sql.DatabaseMetaData#getFunctions's java doc, I
find it will return the system and user functions available in the Catalog.
I think you are right. Therefore, we'd better to rename to the
listFunctions(SessionHandle sessionHandle, OperationHandle operationHandle,
String catalog, String database, ShowFunctionsOperation.FunctionScope) and
it returns FunctionInfo.

>> SQLGatewayService.getGatewayInfo()/getSessionConfig

The result of the SQLGatewayService.getGatewayInfo and getSessionConfig is
not used by the endpoint. The endpoint just serializes the result and
presents it to the users. If we use the ReadableConfig, it's hard for us to
iterate all the key value pairs.

> configure_session VS initialize_session
>> If calling it initialize_session, should we limit it only being called
once?

If we limit it only being called once, it allows the input of the
initialize_session script. But the current design in the Gateway is aligned
with the TableEnvironment#executeSql. That is, the input of the statement
is a single statement rather than the script. Considering the API in the
FLIP is not as same as the initialization in the CLI, I think we can use
the configure_session? What do you think, Timo?

Best,
Shengkai







Timo Walther <tw...@apache.org> 于2022年5月16日周一 14:28写道：

> Hi Shengkai, Hi Jark,
>
> thanks for the additional explanation and the update of the FLIP. This
> will help us in the future for documenting our decisions. The arguments
> why to include the Gateway into the main repo make a lot of sense to me.
> Esp. also because both CLI and gateway need some parsing functionality
> that is dependent on the current state of the SQL syntax.
>
> Here is my last set of feedback, other than that +1 for the proposal:
>
> Serialize the LogicalType
>
> The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is
> different from the current master. We are using the serializable
> representation of LogicalType as much as possible nowadays. We should
> definitely just use LogicalTypeJsonSerializer and not a second JSON
> representation.
>
> 1) Serialize the RowData
>
> Side note for serializing ROWs: we should not use field names in JSON
> object keys. As e.g. `null` and other names with special characters
> cause issues in JSON.
>
> 2) We propose the state machine like HiveServer2
>
> Have you also thought about using Flink's state types from Flink
> tasks/jobs? If we were using Flink types directly, it would be easier to
> monitor the execution of a INSERT INTO job via the gateway without
> having to map state types. Monitoring jobs is the most important
> functionality and should be in sync with regular Flink job monitoring. A
> HiveServer2 endpoint can still perform mapping if necessary, but within
> the Flink code base we use a consistent state transition scheme.
>
> 3) Options to configure the REST endpoint
>
> Given that we also want to be able to put endpoint options into
> flink-conf.yaml, we should make those option a bit more globally
> understandable:
>
> endpoint.type -> sql-gateway.endpoint.type
> endpoint.rest.port -> sql-gateway.endpoint.rest.port
> ...
>
> 4) Naming conversion
>
> This is very nit picking, but please make sure to name the interfaces
> consistently for abbreviations. Currently, we mostly use `SqlInterface`
> instead of `SQLInterface`. So in the FLIP:
>
> SQLGatewayEndpoint -> SqlGatewayEndpoint
> SQLGatewayEndpointFactory -> SqlGatewayEndpointFactory
> e.g. for REST we do it already: RestEndpointVersion
>
> 5) Another smaller shortcomings in the FLIP
>
> SQLGatewayService.getFunction / UserDefinedFunctionInfo
> Was it a conscious decision to not use FunctionIdentifier here? You will
> not be able to list system functions.
>
> SQLGatewayService.getGatewayInfo()/getSessionConfig
> Why not returning ReadableConfig here?
>
> Thanks,
> Timo
>
>
>
>
> On 12.05.22 17:02, Jark Wu wrote:
> > Hi Shengkai,
> >
> > One comment on the configure_session VS initialize_session,
> > I think configure_session is better than initialize_session,
> > because "initialize" sounds like this method can only be invoked once.
> > But I can see this method is useful to be invoked several times to
> > configure
> > the session during the lifecycle of a session. If calling it
> > initialize_session,
> > should we limit it only be called once?
> >
> > Best,
> > Jark
> >
> >
> >
> > On Thu, 12 May 2022 at 22:38, Jark Wu <imjark@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> >     Hi Timo,
> >
> >     I agree with Shengkai that we should keep Gateway in the Flink main
> >     repo.
> >     Here are my thoughts:
> >
> >     1) It is not feasible and maintainable to invoke different versions
> >     of Flink dependencies
> >     in one JVM. I'm afraid it would be a nightmare when the code is
> >     messed up with all the reflections.
> >
> >     2) Actually, it's very mature to support submitting multi-version by
> >     deploying one
> >     gateway process for one version. Many systems are following this
> >     architecture as Shengkai
> >     mentioned above, including the Ververica Platform. And I don't see
> >     any problem with it.
> >
> >     3) As Jingsong said, SQL CLI should build on top of Gateway to share
> >     code and have the ability to
> >       connect to Gateway. If SQL CLI is still kept in the Flink repo, we
> >     have to put Gateway there as well.
> >
> >     4) Besides, Gateway is indispensable for a SQL engine (think of
> >     Trino/Presto, Spark, Hive).
> >     Otherwise, Flink will always be a processing system. With Gateway
> >     inside the Flink repo,
> >     we can provide an out-of-box experience as a SQL query engine. Users
> >     can try out the gateway
> >     for the latest version when a new version is released.
> >
> >     5) Last, Gateway inside the Flink repo can ensure the highest degree
> >     of version compatibility.
> >
> >     Best,
> >     Jark
> >
> >
> >     On Thu, 12 May 2022 at 19:19, Shengkai Fang <fskmine@gmail.com
> >     <ma...@gmail.com>> wrote:
> >
> >         Hi Timo.
> >
> >         Thanks for your feedback!
> >
> >          > It would be great if you can copy/paste some of the tricky
> design
> >         decisions into the FLIP.
> >
> >         Yes. I have already added a section about the `Rejected
> >         Alternatives`. It
> >         includes the discuss about the
> >         - TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
> >         - Support the multi-version Flink in the Gateway VS Support the
> >         multi-version in the external Service
> >         - Merge Gateway into the Flink code base VS Support Gateway in
> >         the another
> >         repo
> >         ...
> >
> >
> >          > Separate code base
> >
> >         Let me summarize all the discussion about the separate code
> >         base. Please
> >         correct me if anything is wrong.
> >
> >
> >         About support for the Gateway in the Flink code base.
> >
> >
> >         1. It reduces the cost including tests, release and document. We
> >         don't need
> >         to upgrade the Gateway when Flink releases.
> >         2. The SQL Client has the ability to submit the SQL to the
> >         Gateway. If we
> >         move the Gateway to the outside repo, the SQL Client doesn't
> >         have the
> >         related dependencies.
> >         3. It is benefit for us to reuse code in the same repo.
> >
> >         About supporting the Gateway in another repo.
> >
> >         1. It is easier to support the multi-version Flink;
> >
> >
> >         First of all, I think we should support the multiple Flink
> >         version with
> >         another component that uses the Gateway from the individual
> >         Flink releases.
> >         Zeppelin, Livy, and Kyuubi all use similar architecture.
> >
> >         The current Gateway in the FLIP is bound to the specific Flink
> >         version,
> >         which needs to compile the SQL and submit the job to the cluster
> >         with the
> >         specified Flink version.
> >         We can introduce a service in the future(Maybe we can name it
> >         MultipleVersionFlinkService? ). It may tell the client where the
> >         Gateway
> >         with specified version is and the client just communicate with
> >         the Gateway
> >         later. It may also just parse the request,  convert the request
> >         to the REST
> >         API in the FLIP and send to the Gateway with specified version.
> >
> >         Therefore, I think we should merge the SQL Gateway inside the
> >         Flink repo.
> >         The MultipleVersionFlinkService may be inside or outside the
> repo.
> >
> >         I add more details in the `Rejected Alternatives` in the FLIP.
> >
> >          > Missing REST API spec
> >
> >         Good point! I add a section in the `Appendix` about the
> >         serialization of
> >         the ResultSet. It includes how to serialize the schema, RowData
> and
> >         exceptions.
> >
> >          > Consistency of concepts
> >
> >         configure_session VS initialize_session
> >
> >         Okay. I think it's better to use the initialize_session.
> >
> >         TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and
> >         CatalogFunction
> >
> >         The CatalogTable and CatalogFunction are much heavier than the
> >         TableInfo
> >         and UserDefinedFunction. The CatalogManager requires reading
> >         from the
> >         Catalog to get the schema. But in the listTables only care about
> >         the table
> >         name, which is much lighter. Therefore, we propose to use the
> >         TableInfo
> >         with required fields.
> >
> >          > Result Retrieval
> >
> >         Yes. Currently the API is only used for the preview. I added a
> >         section in
> >         the `Rejected Alternatives` about this.
> >
> >         Best,
> >         Shengkai
> >
>
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Timo Walther <tw...@apache.org>]

Hi Shengkai, Hi Jark,

thanks for the additional explanation and the update of the FLIP. This 
will help us in the future for documenting our decisions. The arguments 
why to include the Gateway into the main repo make a lot of sense to me. 
Esp. also because both CLI and gateway need some parsing functionality 
that is dependent on the current state of the SQL syntax.

Here is my last set of feedback, other than that +1 for the proposal:

Serialize the LogicalType

The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is 
different from the current master. We are using the serializable 
representation of LogicalType as much as possible nowadays. We should 
definitely just use LogicalTypeJsonSerializer and not a second JSON 
representation.

1) Serialize the RowData

Side note for serializing ROWs: we should not use field names in JSON 
object keys. As e.g. `null` and other names with special characters 
cause issues in JSON.

2) We propose the state machine like HiveServer2

Have you also thought about using Flink's state types from Flink 
tasks/jobs? If we were using Flink types directly, it would be easier to 
monitor the execution of a INSERT INTO job via the gateway without 
having to map state types. Monitoring jobs is the most important 
functionality and should be in sync with regular Flink job monitoring. A 
HiveServer2 endpoint can still perform mapping if necessary, but within 
the Flink code base we use a consistent state transition scheme.

3) Options to configure the REST endpoint

Given that we also want to be able to put endpoint options into 
flink-conf.yaml, we should make those option a bit more globally 
understandable:

endpoint.type -> sql-gateway.endpoint.type
endpoint.rest.port -> sql-gateway.endpoint.rest.port
...

4) Naming conversion

This is very nit picking, but please make sure to name the interfaces 
consistently for abbreviations. Currently, we mostly use `SqlInterface` 
instead of `SQLInterface`. So in the FLIP:

SQLGatewayEndpoint -> SqlGatewayEndpoint
SQLGatewayEndpointFactory -> SqlGatewayEndpointFactory
e.g. for REST we do it already: RestEndpointVersion

5) Another smaller shortcomings in the FLIP

SQLGatewayService.getFunction / UserDefinedFunctionInfo
Was it a conscious decision to not use FunctionIdentifier here? You will 
not be able to list system functions.

SQLGatewayService.getGatewayInfo()/getSessionConfig
Why not returning ReadableConfig here?

Thanks,
Timo




On 12.05.22 17:02, Jark Wu wrote:
> Hi Shengkai,
> 
> One comment on the configure_session VS initialize_session,
> I think configure_session is better than initialize_session,
> because "initialize" sounds like this method can only be invoked once.
> But I can see this method is useful to be invoked several times to 
> configure
> the session during the lifecycle of a session. If calling it 
> initialize_session,
> should we limit it only be called once?
> 
> Best,
> Jark
> 
> 
> 
> On Thu, 12 May 2022 at 22:38, Jark Wu <imjark@gmail.com 
> <ma...@gmail.com>> wrote:
> 
>     Hi Timo,
> 
>     I agree with Shengkai that we should keep Gateway in the Flink main
>     repo.
>     Here are my thoughts:
> 
>     1) It is not feasible and maintainable to invoke different versions
>     of Flink dependencies
>     in one JVM. I'm afraid it would be a nightmare when the code is
>     messed up with all the reflections.
> 
>     2) Actually, it's very mature to support submitting multi-version by
>     deploying one
>     gateway process for one version. Many systems are following this
>     architecture as Shengkai
>     mentioned above, including the Ververica Platform. And I don't see
>     any problem with it.
> 
>     3) As Jingsong said, SQL CLI should build on top of Gateway to share
>     code and have the ability to
>       connect to Gateway. If SQL CLI is still kept in the Flink repo, we
>     have to put Gateway there as well.
> 
>     4) Besides, Gateway is indispensable for a SQL engine (think of
>     Trino/Presto, Spark, Hive).
>     Otherwise, Flink will always be a processing system. With Gateway
>     inside the Flink repo,
>     we can provide an out-of-box experience as a SQL query engine. Users
>     can try out the gateway
>     for the latest version when a new version is released.
> 
>     5) Last, Gateway inside the Flink repo can ensure the highest degree
>     of version compatibility.
> 
>     Best,
>     Jark
> 
> 
>     On Thu, 12 May 2022 at 19:19, Shengkai Fang <fskmine@gmail.com
>     <ma...@gmail.com>> wrote:
> 
>         Hi Timo.
> 
>         Thanks for your feedback!
> 
>          > It would be great if you can copy/paste some of the tricky design
>         decisions into the FLIP.
> 
>         Yes. I have already added a section about the `Rejected
>         Alternatives`. It
>         includes the discuss about the
>         - TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
>         - Support the multi-version Flink in the Gateway VS Support the
>         multi-version in the external Service
>         - Merge Gateway into the Flink code base VS Support Gateway in
>         the another
>         repo
>         ...
> 
> 
>          > Separate code base
> 
>         Let me summarize all the discussion about the separate code
>         base. Please
>         correct me if anything is wrong.
> 
> 
>         About support for the Gateway in the Flink code base.
> 
> 
>         1. It reduces the cost including tests, release and document. We
>         don't need
>         to upgrade the Gateway when Flink releases.
>         2. The SQL Client has the ability to submit the SQL to the
>         Gateway. If we
>         move the Gateway to the outside repo, the SQL Client doesn't
>         have the
>         related dependencies.
>         3. It is benefit for us to reuse code in the same repo.
> 
>         About supporting the Gateway in another repo.
> 
>         1. It is easier to support the multi-version Flink;
> 
> 
>         First of all, I think we should support the multiple Flink
>         version with
>         another component that uses the Gateway from the individual
>         Flink releases.
>         Zeppelin, Livy, and Kyuubi all use similar architecture.
> 
>         The current Gateway in the FLIP is bound to the specific Flink
>         version,
>         which needs to compile the SQL and submit the job to the cluster
>         with the
>         specified Flink version.
>         We can introduce a service in the future(Maybe we can name it
>         MultipleVersionFlinkService? ). It may tell the client where the
>         Gateway
>         with specified version is and the client just communicate with
>         the Gateway
>         later. It may also just parse the request,  convert the request
>         to the REST
>         API in the FLIP and send to the Gateway with specified version.
> 
>         Therefore, I think we should merge the SQL Gateway inside the
>         Flink repo.
>         The MultipleVersionFlinkService may be inside or outside the repo.
> 
>         I add more details in the `Rejected Alternatives` in the FLIP.
> 
>          > Missing REST API spec
> 
>         Good point! I add a section in the `Appendix` about the
>         serialization of
>         the ResultSet. It includes how to serialize the schema, RowData and
>         exceptions.
> 
>          > Consistency of concepts
> 
>         configure_session VS initialize_session
> 
>         Okay. I think it's better to use the initialize_session.
> 
>         TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and
>         CatalogFunction
> 
>         The CatalogTable and CatalogFunction are much heavier than the
>         TableInfo
>         and UserDefinedFunction. The CatalogManager requires reading
>         from the
>         Catalog to get the schema. But in the listTables only care about
>         the table
>         name, which is much lighter. Therefore, we propose to use the
>         TableInfo
>         with required fields.
> 
>          > Result Retrieval
> 
>         Yes. Currently the API is only used for the preview. I added a
>         section in
>         the `Rejected Alternatives` about this.
> 
>         Best,
>         Shengkai
> 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jark Wu <im...@gmail.com>]

Hi Shengkai,

One comment on the configure_session VS initialize_session,
I think configure_session is better than initialize_session,
because "initialize" sounds like this method can only be invoked once.
But I can see this method is useful to be invoked several times to
configure
the session during the lifecycle of a session. If calling it
initialize_session,
should we limit it only be called once?

Best,
Jark



On Thu, 12 May 2022 at 22:38, Jark Wu <im...@gmail.com> wrote:

> Hi Timo,
>
> I agree with Shengkai that we should keep Gateway in the Flink main repo.
> Here are my thoughts:
>
> 1) It is not feasible and maintainable to invoke different versions of
> Flink dependencies
> in one JVM. I'm afraid it would be a nightmare when the code is messed up
> with all the reflections.
>
> 2) Actually, it's very mature to support submitting multi-version by
> deploying one
> gateway process for one version. Many systems are following this
> architecture as Shengkai
> mentioned above, including the Ververica Platform. And I don't see any
> problem with it.
>
> 3) As Jingsong said, SQL CLI should build on top of Gateway to share code
> and have the ability to
>  connect to Gateway. If SQL CLI is still kept in the Flink repo, we have
> to put Gateway there as well.
>
> 4) Besides, Gateway is indispensable for a SQL engine (think of
> Trino/Presto, Spark, Hive).
> Otherwise, Flink will always be a processing system. With Gateway inside
> the Flink repo,
> we can provide an out-of-box experience as a SQL query engine. Users can
> try out the gateway
> for the latest version when a new version is released.
>
> 5) Last, Gateway inside the Flink repo can ensure the highest degree of
> version compatibility.
>
> Best,
> Jark
>
>
> On Thu, 12 May 2022 at 19:19, Shengkai Fang <fs...@gmail.com> wrote:
>
>> Hi Timo.
>>
>> Thanks for your feedback!
>>
>> > It would be great if you can copy/paste some of the tricky design
>> decisions into the FLIP.
>>
>> Yes. I have already added a section about the `Rejected Alternatives`. It
>> includes the discuss about the
>> - TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
>> - Support the multi-version Flink in the Gateway VS Support the
>> multi-version in the external Service
>> - Merge Gateway into the Flink code base VS Support Gateway in the another
>> repo
>> ...
>>
>>
>> > Separate code base
>>
>> Let me summarize all the discussion about the separate code base. Please
>> correct me if anything is wrong.
>>
>>
>> About support for the Gateway in the Flink code base.
>>
>>
>> 1. It reduces the cost including tests, release and document. We don't
>> need
>> to upgrade the Gateway when Flink releases.
>> 2. The SQL Client has the ability to submit the SQL to the Gateway. If we
>> move the Gateway to the outside repo, the SQL Client doesn't have the
>> related dependencies.
>> 3. It is benefit for us to reuse code in the same repo.
>>
>> About supporting the Gateway in another repo.
>>
>> 1. It is easier to support the multi-version Flink;
>>
>>
>> First of all, I think we should support the multiple Flink version with
>> another component that uses the Gateway from the individual Flink
>> releases.
>> Zeppelin, Livy, and Kyuubi all use similar architecture.
>>
>> The current Gateway in the FLIP is bound to the specific Flink version,
>> which needs to compile the SQL and submit the job to the cluster with the
>> specified Flink version.
>> We can introduce a service in the future(Maybe we can name it
>> MultipleVersionFlinkService? ). It may tell the client where the Gateway
>> with specified version is and the client just communicate with the Gateway
>> later. It may also just parse the request,  convert the request to the
>> REST
>> API in the FLIP and send to the Gateway with specified version.
>>
>> Therefore, I think we should merge the SQL Gateway inside the Flink repo.
>> The MultipleVersionFlinkService may be inside or outside the repo.
>>
>> I add more details in the `Rejected Alternatives` in the FLIP.
>>
>> > Missing REST API spec
>>
>> Good point! I add a section in the `Appendix` about the serialization of
>> the ResultSet. It includes how to serialize the schema, RowData and
>> exceptions.
>>
>> > Consistency of concepts
>>
>> configure_session VS initialize_session
>>
>> Okay. I think it's better to use the initialize_session.
>>
>> TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and CatalogFunction
>>
>> The CatalogTable and CatalogFunction are much heavier than the TableInfo
>> and UserDefinedFunction. The CatalogManager requires reading from the
>> Catalog to get the schema. But in the listTables only care about the table
>> name, which is much lighter. Therefore, we propose to use the TableInfo
>> with required fields.
>>
>> > Result Retrieval
>>
>> Yes. Currently the API is only used for the preview. I added a section in
>> the `Rejected Alternatives` about this.
>>
>> Best,
>> Shengkai
>>
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jark Wu <im...@gmail.com>]

Hi Timo,

I agree with Shengkai that we should keep Gateway in the Flink main repo.
Here are my thoughts:

1) It is not feasible and maintainable to invoke different versions of
Flink dependencies
in one JVM. I'm afraid it would be a nightmare when the code is messed up
with all the reflections.

2) Actually, it's very mature to support submitting multi-version by
deploying one
gateway process for one version. Many systems are following this
architecture as Shengkai
mentioned above, including the Ververica Platform. And I don't see any
problem with it.

3) As Jingsong said, SQL CLI should build on top of Gateway to share code
and have the ability to
 connect to Gateway. If SQL CLI is still kept in the Flink repo, we have to
put Gateway there as well.

4) Besides, Gateway is indispensable for a SQL engine (think of
Trino/Presto, Spark, Hive).
Otherwise, Flink will always be a processing system. With Gateway inside
the Flink repo,
we can provide an out-of-box experience as a SQL query engine. Users can
try out the gateway
for the latest version when a new version is released.

5) Last, Gateway inside the Flink repo can ensure the highest degree of
version compatibility.

Best,
Jark


On Thu, 12 May 2022 at 19:19, Shengkai Fang <fs...@gmail.com> wrote:

> Hi Timo.
>
> Thanks for your feedback!
>
> > It would be great if you can copy/paste some of the tricky design
> decisions into the FLIP.
>
> Yes. I have already added a section about the `Rejected Alternatives`. It
> includes the discuss about the
> - TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
> - Support the multi-version Flink in the Gateway VS Support the
> multi-version in the external Service
> - Merge Gateway into the Flink code base VS Support Gateway in the another
> repo
> ...
>
>
> > Separate code base
>
> Let me summarize all the discussion about the separate code base. Please
> correct me if anything is wrong.
>
>
> About support for the Gateway in the Flink code base.
>
>
> 1. It reduces the cost including tests, release and document. We don't need
> to upgrade the Gateway when Flink releases.
> 2. The SQL Client has the ability to submit the SQL to the Gateway. If we
> move the Gateway to the outside repo, the SQL Client doesn't have the
> related dependencies.
> 3. It is benefit for us to reuse code in the same repo.
>
> About supporting the Gateway in another repo.
>
> 1. It is easier to support the multi-version Flink;
>
>
> First of all, I think we should support the multiple Flink version with
> another component that uses the Gateway from the individual Flink releases.
> Zeppelin, Livy, and Kyuubi all use similar architecture.
>
> The current Gateway in the FLIP is bound to the specific Flink version,
> which needs to compile the SQL and submit the job to the cluster with the
> specified Flink version.
> We can introduce a service in the future(Maybe we can name it
> MultipleVersionFlinkService? ). It may tell the client where the Gateway
> with specified version is and the client just communicate with the Gateway
> later. It may also just parse the request,  convert the request to the REST
> API in the FLIP and send to the Gateway with specified version.
>
> Therefore, I think we should merge the SQL Gateway inside the Flink repo.
> The MultipleVersionFlinkService may be inside or outside the repo.
>
> I add more details in the `Rejected Alternatives` in the FLIP.
>
> > Missing REST API spec
>
> Good point! I add a section in the `Appendix` about the serialization of
> the ResultSet. It includes how to serialize the schema, RowData and
> exceptions.
>
> > Consistency of concepts
>
> configure_session VS initialize_session
>
> Okay. I think it's better to use the initialize_session.
>
> TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and CatalogFunction
>
> The CatalogTable and CatalogFunction are much heavier than the TableInfo
> and UserDefinedFunction. The CatalogManager requires reading from the
> Catalog to get the schema. But in the listTables only care about the table
> name, which is much lighter. Therefore, we propose to use the TableInfo
> with required fields.
>
> > Result Retrieval
>
> Yes. Currently the API is only used for the preview. I added a section in
> the `Rejected Alternatives` about this.
>
> Best,
> Shengkai
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi Timo.

Thanks for your feedback!

> It would be great if you can copy/paste some of the tricky design
decisions into the FLIP.

Yes. I have already added a section about the `Rejected Alternatives`. It
includes the discuss about the
- TableInfo and FunctionInfo VS CatalogTable and CatalogFunction
- Support the multi-version Flink in the Gateway VS Support the
multi-version in the external Service
- Merge Gateway into the Flink code base VS Support Gateway in the another
repo
...


> Separate code base

Let me summarize all the discussion about the separate code base. Please
correct me if anything is wrong.


About support for the Gateway in the Flink code base.


1. It reduces the cost including tests, release and document. We don't need
to upgrade the Gateway when Flink releases.
2. The SQL Client has the ability to submit the SQL to the Gateway. If we
move the Gateway to the outside repo, the SQL Client doesn't have the
related dependencies.
3. It is benefit for us to reuse code in the same repo.

About supporting the Gateway in another repo.

1. It is easier to support the multi-version Flink;


First of all, I think we should support the multiple Flink version with
another component that uses the Gateway from the individual Flink releases.
Zeppelin, Livy, and Kyuubi all use similar architecture.

The current Gateway in the FLIP is bound to the specific Flink version,
which needs to compile the SQL and submit the job to the cluster with the
specified Flink version.
We can introduce a service in the future(Maybe we can name it
MultipleVersionFlinkService? ). It may tell the client where the Gateway
with specified version is and the client just communicate with the Gateway
later. It may also just parse the request,  convert the request to the REST
API in the FLIP and send to the Gateway with specified version.

Therefore, I think we should merge the SQL Gateway inside the Flink repo.
The MultipleVersionFlinkService may be inside or outside the repo.

I add more details in the `Rejected Alternatives` in the FLIP.

> Missing REST API spec

Good point! I add a section in the `Appendix` about the serialization of
the ResultSet. It includes how to serialize the schema, RowData and
exceptions.

> Consistency of concepts

configure_session VS initialize_session

Okay. I think it's better to use the initialize_session.

TableInfo + UserDefinedFunctionInfo  VS  CatalogTable and CatalogFunction

The CatalogTable and CatalogFunction are much heavier than the TableInfo
and UserDefinedFunction. The CatalogManager requires reading from the
Catalog to get the schema. But in the listTables only care about the table
name, which is much lighter. Therefore, we propose to use the TableInfo
with required fields.

> Result Retrieval

Yes. Currently the API is only used for the preview. I added a section in
the `Rejected Alternatives` about this.

Best,
Shengkai

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Timo Walther <tw...@apache.org>]

Hi Shengkai,

sorry for jumping in so late into the discussion. There was a lot going 
on in the last 3 weeks. This FLIP is very important and it is great that 
you tackle this topic which is planned since we started with the SQL Client.

The ML discussion is already quite long. It would be great if you can 
copy/paste some of the tricky design decisions into the FLIP. With pros 
and cons into a `Rejected Alternatives` section. Also the `Future work` 
setion does not mention your plans of `merge SQL gateway into 
JobManager`. So far this FLIP seems incomplete to me and a lot of 
important information is actually contained in the discussion thread 
instead of the design document.

For example, Godfrey's comment `Some LogicalTypes can not be serialized, 
such as: CharType(0)` was my thought as well when reading the FLIP 
today. Now I see that you actually answered this with `We can follow the 
behaviour in the LogicalTypeJsonSerializer`. Please update the FLIP for 
these feedback. If people have questions that the FLIP does not answer, 
then the FLIP should be updated. It is totally fine to add more sections 
and structure it further, look at how FLIP-196 evolved because many 
people had interest in this feature.

There is a lot of feedback from my side on this FLIP, some comments are 
nit picking, some comments are crucial. I will start with the crucial 
stuff first, and post my nit comments in a follow up discussion after 
the FLIP had another iteration.

1) Separate code base

I would also vote for a separate code base. The gateway should be able 
to submit to different Flink versions. And by looking at the interfaces 
of `SQLGatewayService` it looks like we are trying to abstract away the 
TableEnvironment, so an endpoint does not depend on internal Flink API. 
Thus, I don't think that frequently changing API should be an issue. We 
just need to design the `SQLGatewayService` per Flink version or with a 
shim layer. Also because the API is already quite stable with 1.15 and 
I'm sure we can mark more interfaces as @Public if that would help.

2) Missing REST API spec

The current FLIP skips some spec for REST API. For example, how do we 
serialize RowData in the REST API? We should have consensus here. The 
format could be in sync with built-in JSON_STRING()? Maybe it would also 
make sense to have 2 data formats (JSON + or serialized with our 
internal serializers?).

3) Consistency of concepts

The concept of an `init statement` should not be exclusive to the CLI 
client. It should be a concept of the gateway as well. Otherwise this 
could become quite complex if some statements can be used in `-i` but 
not in `/configure_session`. Also why not calling this 
`initialize_session` to be consistent.

TableInfo + UserDefinedFunctionInfo seem redundant to me, they have the 
same purpose as CatalogTable and CatalogFunction. What is TableKind.ALL?

4) Result Retrieval

Shall we make a difference between result retrieval for interactive 
sessions (just previews) vs. result retrieval for fault tolerant 
exactly-once results? The current design seems to be made for the first 
use case. A JDBC abstraction will not be able to provide consistent 
results in case of a failure. Maybe we should mention this in the FLIP?

Thanks,
Timo

Am 10.05.22 um 09:20 schrieb Jark Wu:
> Thank Shengkai for driving this work!
> +1 to start a VOTE.
>
> Best,
> Jark
>
> On Tue, 10 May 2022 at 12:13, Shengkai Fang <fs...@gmail.com> wrote:
>
>> Hi, everyone.
>>
>> Thanks for all the inputs. Hope my feedback can address most of questions.
>>
>> After the long discussion, I think we have reached the consensus about the
>> design. If the discussion doesn't get more response, I think we can start
>> the vote tomorrow.
>>
>> Best,
>> Shengkai
>>
>> Shengkai Fang <fs...@gmail.com> 于2022年5月9日周一 14:18写道：
>>
>>> Hi JingSong.
>>>
>>> Thanks for your feedback.
>>>
>>>> reorganize the FLIP, what Pluggable Endpoint Discovery is, and how
>> users
>>> to add new Endpoint, before introducing SQLGatewayService.
>>>
>>> I update the FLIP:  reorganize the order and add more details.
>>>
>>>> Then I have some doubts about the name SQLGatewayService, I think
>>>> Service is seriously misleading, it is just an implementation class
>>>> and not a Service.
>>> After discuss with Jingsong offline, we agree to still use the
>>> SQLGatewayService. According to the defination of the wiki[1], service
>>> refers to a software functionality or a set of software functionalities
>>> with a purpose that different clients can reuse for different purposes.
>>> Here the GatewayService plays the same role.
>>>
>>> Best,
>>> Shengkai
>>>
>>> [1] https://en.wikipedia.org/wiki/Service_(systems_architecture)
>>>
>>> Jingsong Li <ji...@gmail.com> 于2022年5月7日周六 16:55写道：
>>>
>>>> Hi Shengkai, thanks for your reply.
>>>>
>>>>> REST API is the user interface. The REST API transforms the request to
>>>> the
>>>> invocation of the SQLGatewayService that is the one doing the work. We
>>>> split the Gateway into the SQLGatewayService and Endpoint(REST API) and
>>>> its
>>>> benefit is that all the Endpoints share the same SQLGatewayService.
>>>>
>>>> Finally, I got the point of SQLGatewayService, It is just for
>>>> `Pluggable Endpoint Discovery`.
>>>> I suggest you reorganize the FLIP, what Pluggable Endpoint Discovery
>>>> is, and how users to add new Endpoint, before introducing
>>>> SQLGatewayService.
>>>>
>>>> Then I have some doubts about the name SQLGatewayService, I think
>>>> Service is seriously misleading, it is just an implementation class
>>>> and not a Service.
>>>>
>>>> What about just `SQLGateway`?
>>>>
>>>> Best,
>>>> Jingsong
>>>>
>>>> On Sat, May 7, 2022 at 4:03 PM Shengkai Fang <fs...@gmail.com> wrote:
>>>>> Hi Martijn.
>>>>>
>>>>> It seems we have reached consensus to support the Gateway inside the
>>>> Flink
>>>>> code base.
>>>>>
>>>>> Hi, Echo.
>>>>>
>>>>> Thanks for your interest.
>>>>>
>>>>>> whether flink-sql-gateway should be retained in the Flink project.
>>>>> I think the discussion above is clear. It is the essential tool to
>>>> provide
>>>>> out-of-box experience for users.
>>>>>
>>>>>> For stream processing, what's the point of getting the result? Is it
>>>> just
>>>>> for debugging and how to unify with batch processing
>>>>>
>>>>> At the client side, when the OperationStaus is FINISHED, the client is
>>>> able
>>>>> to fetch the results from the Gateway. It is unified with the batch
>>>>> processing now.
>>>>>
>>>>>> For batch processing, does the gateway need to cache all fetch
>>>> results?
>>>>> No. In the Gateway, we will only buffer partial data and wait for the
>>>>> client to consume. If the client takes away the buffered data, the
>>>> Gateway
>>>>> will clear the buffer and notify the fetcher thread starts to work
>> until
>>>>> the buffer is full again. The mechanism behind is much like the
>>>>> producer-consumer model[1].
>>>>>
>>>>> [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
>>>>>
>>>>>> Whether executing query and fetch results should be synchronous or
>>>>> asynchronous?
>>>>>
>>>>> Do you mean the executeStatement response should also contain the
>>>> result? I
>>>>> don't think using the asynchronous API will cause performance
>>>> regression.
>>>>> In most cases, the network latency is about 100ms or lower. You can
>> ping
>>>>> www.baidu.com or www.google.com to test the latency.
>>>>>
>>>>>> When executing a query in flink-sql-client, I often find error logs
>> of
>>>>> FlinkJobNotFoundException. Should this be optimized?
>>>>>
>>>>> It's related to the current client implementation. I think you can
>> open
>>>> a
>>>>> jira ticket, add more details and we can discuss the problem in the
>>>> ticket.
>>>>> In the FLIP-91, the Gateway can store the log per Operation. It may
>>>> solve
>>>>> your problems.
>>>>>
>>>>>
>>>>> Best,
>>>>> Shengkai

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jark Wu <im...@gmail.com>]

Thank Shengkai for driving this work!
+1 to start a VOTE.

Best,
Jark

On Tue, 10 May 2022 at 12:13, Shengkai Fang <fs...@gmail.com> wrote:

> Hi, everyone.
>
> Thanks for all the inputs. Hope my feedback can address most of questions.
>
> After the long discussion, I think we have reached the consensus about the
> design. If the discussion doesn't get more response, I think we can start
> the vote tomorrow.
>
> Best,
> Shengkai
>
> Shengkai Fang <fs...@gmail.com> 于2022年5月9日周一 14:18写道：
>
> > Hi JingSong.
> >
> > Thanks for your feedback.
> >
> > > reorganize the FLIP, what Pluggable Endpoint Discovery is, and how
> users
> > to add new Endpoint, before introducing SQLGatewayService.
> >
> > I update the FLIP:  reorganize the order and add more details.
> >
> > > Then I have some doubts about the name SQLGatewayService, I think
> > > Service is seriously misleading, it is just an implementation class
> > > and not a Service.
> >
> > After discuss with Jingsong offline, we agree to still use the
> > SQLGatewayService. According to the defination of the wiki[1], service
> > refers to a software functionality or a set of software functionalities
> > with a purpose that different clients can reuse for different purposes.
> > Here the GatewayService plays the same role.
> >
> > Best,
> > Shengkai
> >
> > [1] https://en.wikipedia.org/wiki/Service_(systems_architecture)
> >
> > Jingsong Li <ji...@gmail.com> 于2022年5月7日周六 16:55写道：
> >
> >> Hi Shengkai, thanks for your reply.
> >>
> >> > REST API is the user interface. The REST API transforms the request to
> >> the
> >> invocation of the SQLGatewayService that is the one doing the work. We
> >> split the Gateway into the SQLGatewayService and Endpoint(REST API) and
> >> its
> >> benefit is that all the Endpoints share the same SQLGatewayService.
> >>
> >> Finally, I got the point of SQLGatewayService, It is just for
> >> `Pluggable Endpoint Discovery`.
> >> I suggest you reorganize the FLIP, what Pluggable Endpoint Discovery
> >> is, and how users to add new Endpoint, before introducing
> >> SQLGatewayService.
> >>
> >> Then I have some doubts about the name SQLGatewayService, I think
> >> Service is seriously misleading, it is just an implementation class
> >> and not a Service.
> >>
> >> What about just `SQLGateway`?
> >>
> >> Best,
> >> Jingsong
> >>
> >> On Sat, May 7, 2022 at 4:03 PM Shengkai Fang <fs...@gmail.com> wrote:
> >> >
> >> > Hi Martijn.
> >> >
> >> > It seems we have reached consensus to support the Gateway inside the
> >> Flink
> >> > code base.
> >> >
> >> > Hi, Echo.
> >> >
> >> > Thanks for your interest.
> >> >
> >> > > whether flink-sql-gateway should be retained in the Flink project.
> >> >
> >> > I think the discussion above is clear. It is the essential tool to
> >> provide
> >> > out-of-box experience for users.
> >> >
> >> > > For stream processing, what's the point of getting the result? Is it
> >> just
> >> > for debugging and how to unify with batch processing
> >> >
> >> > At the client side, when the OperationStaus is FINISHED, the client is
> >> able
> >> > to fetch the results from the Gateway. It is unified with the batch
> >> > processing now.
> >> >
> >> > > For batch processing, does the gateway need to cache all fetch
> >> results?
> >> >
> >> > No. In the Gateway, we will only buffer partial data and wait for the
> >> > client to consume. If the client takes away the buffered data, the
> >> Gateway
> >> > will clear the buffer and notify the fetcher thread starts to work
> until
> >> > the buffer is full again. The mechanism behind is much like the
> >> > producer-consumer model[1].
> >> >
> >> > [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
> >> >
> >> > > Whether executing query and fetch results should be synchronous or
> >> > asynchronous?
> >> >
> >> > Do you mean the executeStatement response should also contain the
> >> result? I
> >> > don't think using the asynchronous API will cause performance
> >> regression.
> >> > In most cases, the network latency is about 100ms or lower. You can
> ping
> >> > www.baidu.com or www.google.com to test the latency.
> >> >
> >> > > When executing a query in flink-sql-client, I often find error logs
> of
> >> > FlinkJobNotFoundException. Should this be optimized?
> >> >
> >> > It's related to the current client implementation. I think you can
> open
> >> a
> >> > jira ticket, add more details and we can discuss the problem in the
> >> ticket.
> >> > In the FLIP-91, the Gateway can store the log per Operation. It may
> >> solve
> >> > your problems.
> >> >
> >> >
> >> > Best,
> >> > Shengkai
> >>
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi, everyone.

Thanks for all the inputs. Hope my feedback can address most of questions.

After the long discussion, I think we have reached the consensus about the
design. If the discussion doesn't get more response, I think we can start
the vote tomorrow.

Best,
Shengkai

Shengkai Fang <fs...@gmail.com> 于2022年5月9日周一 14:18写道：

> Hi JingSong.
>
> Thanks for your feedback.
>
> > reorganize the FLIP, what Pluggable Endpoint Discovery is, and how users
> to add new Endpoint, before introducing SQLGatewayService.
>
> I update the FLIP:  reorganize the order and add more details.
>
> > Then I have some doubts about the name SQLGatewayService, I think
> > Service is seriously misleading, it is just an implementation class
> > and not a Service.
>
> After discuss with Jingsong offline, we agree to still use the
> SQLGatewayService. According to the defination of the wiki[1], service
> refers to a software functionality or a set of software functionalities
> with a purpose that different clients can reuse for different purposes.
> Here the GatewayService plays the same role.
>
> Best,
> Shengkai
>
> [1] https://en.wikipedia.org/wiki/Service_(systems_architecture)
>
> Jingsong Li <ji...@gmail.com> 于2022年5月7日周六 16:55写道：
>
>> Hi Shengkai, thanks for your reply.
>>
>> > REST API is the user interface. The REST API transforms the request to
>> the
>> invocation of the SQLGatewayService that is the one doing the work. We
>> split the Gateway into the SQLGatewayService and Endpoint(REST API) and
>> its
>> benefit is that all the Endpoints share the same SQLGatewayService.
>>
>> Finally, I got the point of SQLGatewayService, It is just for
>> `Pluggable Endpoint Discovery`.
>> I suggest you reorganize the FLIP, what Pluggable Endpoint Discovery
>> is, and how users to add new Endpoint, before introducing
>> SQLGatewayService.
>>
>> Then I have some doubts about the name SQLGatewayService, I think
>> Service is seriously misleading, it is just an implementation class
>> and not a Service.
>>
>> What about just `SQLGateway`?
>>
>> Best,
>> Jingsong
>>
>> On Sat, May 7, 2022 at 4:03 PM Shengkai Fang <fs...@gmail.com> wrote:
>> >
>> > Hi Martijn.
>> >
>> > It seems we have reached consensus to support the Gateway inside the
>> Flink
>> > code base.
>> >
>> > Hi, Echo.
>> >
>> > Thanks for your interest.
>> >
>> > > whether flink-sql-gateway should be retained in the Flink project.
>> >
>> > I think the discussion above is clear. It is the essential tool to
>> provide
>> > out-of-box experience for users.
>> >
>> > > For stream processing, what's the point of getting the result? Is it
>> just
>> > for debugging and how to unify with batch processing
>> >
>> > At the client side, when the OperationStaus is FINISHED, the client is
>> able
>> > to fetch the results from the Gateway. It is unified with the batch
>> > processing now.
>> >
>> > > For batch processing, does the gateway need to cache all fetch
>> results?
>> >
>> > No. In the Gateway, we will only buffer partial data and wait for the
>> > client to consume. If the client takes away the buffered data, the
>> Gateway
>> > will clear the buffer and notify the fetcher thread starts to work until
>> > the buffer is full again. The mechanism behind is much like the
>> > producer-consumer model[1].
>> >
>> > [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
>> >
>> > > Whether executing query and fetch results should be synchronous or
>> > asynchronous?
>> >
>> > Do you mean the executeStatement response should also contain the
>> result? I
>> > don't think using the asynchronous API will cause performance
>> regression.
>> > In most cases, the network latency is about 100ms or lower. You can ping
>> > www.baidu.com or www.google.com to test the latency.
>> >
>> > > When executing a query in flink-sql-client, I often find error logs of
>> > FlinkJobNotFoundException. Should this be optimized?
>> >
>> > It's related to the current client implementation. I think you can open
>> a
>> > jira ticket, add more details and we can discuss the problem in the
>> ticket.
>> > In the FLIP-91, the Gateway can store the log per Operation. It may
>> solve
>> > your problems.
>> >
>> >
>> > Best,
>> > Shengkai
>>
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi JingSong.

Thanks for your feedback.

> reorganize the FLIP, what Pluggable Endpoint Discovery is, and how users
to add new Endpoint, before introducing SQLGatewayService.

I update the FLIP:  reorganize the order and add more details.

> Then I have some doubts about the name SQLGatewayService, I think
> Service is seriously misleading, it is just an implementation class
> and not a Service.

After discuss with Jingsong offline, we agree to still use the
SQLGatewayService. According to the defination of the wiki[1], service
refers to a software functionality or a set of software functionalities
with a purpose that different clients can reuse for different purposes.
Here the GatewayService plays the same role.

Best,
Shengkai

[1] https://en.wikipedia.org/wiki/Service_(systems_architecture)

Jingsong Li <ji...@gmail.com> 于2022年5月7日周六 16:55写道：

> Hi Shengkai, thanks for your reply.
>
> > REST API is the user interface. The REST API transforms the request to
> the
> invocation of the SQLGatewayService that is the one doing the work. We
> split the Gateway into the SQLGatewayService and Endpoint(REST API) and its
> benefit is that all the Endpoints share the same SQLGatewayService.
>
> Finally, I got the point of SQLGatewayService, It is just for
> `Pluggable Endpoint Discovery`.
> I suggest you reorganize the FLIP, what Pluggable Endpoint Discovery
> is, and how users to add new Endpoint, before introducing
> SQLGatewayService.
>
> Then I have some doubts about the name SQLGatewayService, I think
> Service is seriously misleading, it is just an implementation class
> and not a Service.
>
> What about just `SQLGateway`?
>
> Best,
> Jingsong
>
> On Sat, May 7, 2022 at 4:03 PM Shengkai Fang <fs...@gmail.com> wrote:
> >
> > Hi Martijn.
> >
> > It seems we have reached consensus to support the Gateway inside the
> Flink
> > code base.
> >
> > Hi, Echo.
> >
> > Thanks for your interest.
> >
> > > whether flink-sql-gateway should be retained in the Flink project.
> >
> > I think the discussion above is clear. It is the essential tool to
> provide
> > out-of-box experience for users.
> >
> > > For stream processing, what's the point of getting the result? Is it
> just
> > for debugging and how to unify with batch processing
> >
> > At the client side, when the OperationStaus is FINISHED, the client is
> able
> > to fetch the results from the Gateway. It is unified with the batch
> > processing now.
> >
> > > For batch processing, does the gateway need to cache all fetch results?
> >
> > No. In the Gateway, we will only buffer partial data and wait for the
> > client to consume. If the client takes away the buffered data, the
> Gateway
> > will clear the buffer and notify the fetcher thread starts to work until
> > the buffer is full again. The mechanism behind is much like the
> > producer-consumer model[1].
> >
> > [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
> >
> > > Whether executing query and fetch results should be synchronous or
> > asynchronous?
> >
> > Do you mean the executeStatement response should also contain the
> result? I
> > don't think using the asynchronous API will cause performance regression.
> > In most cases, the network latency is about 100ms or lower. You can ping
> > www.baidu.com or www.google.com to test the latency.
> >
> > > When executing a query in flink-sql-client, I often find error logs of
> > FlinkJobNotFoundException. Should this be optimized?
> >
> > It's related to the current client implementation. I think you can open a
> > jira ticket, add more details and we can discuss the problem in the
> ticket.
> > In the FLIP-91, the Gateway can store the log per Operation. It may solve
> > your problems.
> >
> >
> > Best,
> > Shengkai
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jingsong Li <ji...@gmail.com>]

Hi Shengkai, thanks for your reply.

> REST API is the user interface. The REST API transforms the request to the
invocation of the SQLGatewayService that is the one doing the work. We
split the Gateway into the SQLGatewayService and Endpoint(REST API) and its
benefit is that all the Endpoints share the same SQLGatewayService.

Finally, I got the point of SQLGatewayService, It is just for
`Pluggable Endpoint Discovery`.
I suggest you reorganize the FLIP, what Pluggable Endpoint Discovery
is, and how users to add new Endpoint, before introducing
SQLGatewayService.

Then I have some doubts about the name SQLGatewayService, I think
Service is seriously misleading, it is just an implementation class
and not a Service.

What about just `SQLGateway`?

Best,
Jingsong

On Sat, May 7, 2022 at 4:03 PM Shengkai Fang <fs...@gmail.com> wrote:
>
> Hi Martijn.
>
> It seems we have reached consensus to support the Gateway inside the Flink
> code base.
>
> Hi, Echo.
>
> Thanks for your interest.
>
> > whether flink-sql-gateway should be retained in the Flink project.
>
> I think the discussion above is clear. It is the essential tool to provide
> out-of-box experience for users.
>
> > For stream processing, what's the point of getting the result? Is it just
> for debugging and how to unify with batch processing
>
> At the client side, when the OperationStaus is FINISHED, the client is able
> to fetch the results from the Gateway. It is unified with the batch
> processing now.
>
> > For batch processing, does the gateway need to cache all fetch results?
>
> No. In the Gateway, we will only buffer partial data and wait for the
> client to consume. If the client takes away the buffered data, the Gateway
> will clear the buffer and notify the fetcher thread starts to work until
> the buffer is full again. The mechanism behind is much like the
> producer-consumer model[1].
>
> [1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem
>
> > Whether executing query and fetch results should be synchronous or
> asynchronous?
>
> Do you mean the executeStatement response should also contain the result? I
> don't think using the asynchronous API will cause performance regression.
> In most cases, the network latency is about 100ms or lower. You can ping
> www.baidu.com or www.google.com to test the latency.
>
> > When executing a query in flink-sql-client, I often find error logs of
> FlinkJobNotFoundException. Should this be optimized?
>
> It's related to the current client implementation. I think you can open a
> jira ticket, add more details and we can discuss the problem in the ticket.
> In the FLIP-91, the Gateway can store the log per Operation. It may solve
> your problems.
>
>
> Best,
> Shengkai

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi Martijn.

It seems we have reached consensus to support the Gateway inside the Flink
code base.

Hi, Echo.

Thanks for your interest.

> whether flink-sql-gateway should be retained in the Flink project.

I think the discussion above is clear. It is the essential tool to provide
out-of-box experience for users.

> For stream processing, what's the point of getting the result? Is it just
for debugging and how to unify with batch processing

At the client side, when the OperationStaus is FINISHED, the client is able
to fetch the results from the Gateway. It is unified with the batch
processing now.

> For batch processing, does the gateway need to cache all fetch results?

No. In the Gateway, we will only buffer partial data and wait for the
client to consume. If the client takes away the buffered data, the Gateway
will clear the buffer and notify the fetcher thread starts to work until
the buffer is full again. The mechanism behind is much like the
producer-consumer model[1].

[1] https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem

> Whether executing query and fetch results should be synchronous or
asynchronous?

Do you mean the executeStatement response should also contain the result? I
don't think using the asynchronous API will cause performance regression.
In most cases, the network latency is about 100ms or lower. You can ping
www.baidu.com or www.google.com to test the latency.

> When executing a query in flink-sql-client, I often find error logs of
FlinkJobNotFoundException. Should this be optimized?

It's related to the current client implementation. I think you can open a
jira ticket, add more details and we can discuss the problem in the ticket.
In the FLIP-91, the Gateway can store the log per Operation. It may solve
your problems.


Best,
Shengkai

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Echo Lee <li...@icloud.com.INVALID>]

Hi Shengkai,

Thanks for driving the proposal, I've been paying attention to the ververica/flink-sql-gateway[1] project recently. Because the Flink version the company is currently using is 1.14, so try to upgrade the gateway to Flink-1.14, because of the changes in flink-table api makes the upgrade very painful. At the same time, it is also found that flink-sql-gateway and flink-sql-client have many similarities in design. So at present, I am more concerned about whether flink-sql-gateway should be independent or retained in the Flink project similar to sql-client.
In addition, I am also very interested in flink-sql-gateway's support for OLAP. Recently, when I tried to upgrade flink-sql-gateway, I was confused about the query statement to get the result. There are mainly the following issues:
1. For stream processing, what's the point of getting the result? Is it just for debugging and how to unify with batch processing
2. For batch processing, does the gateway need to cache all fetch results?
3. Whether executing query and fetch results should be synchronous or asynchronous?
4. When executing query in flink-sql-client, I often find error logs of FlinkJobNotFoundException. Should this be optimized?

[1] https://github.com/ververica/flink-sql-gateway

Best,
Echo Lee

> 在 2022年5月6日，下午5:05，LuNing Wang <wa...@gmail.com> 写道：
> 
> Thanks, Shengkai for driving.  And all for your discussion.
> 
> 
> 
>> intergate the Gateway into the Flink code base
> 
> After I talk with Shengkai offline and read the topic `Powering HTAP at
> ByteDance with Apache Flink` of Flink Forward Asia. I think it is better to
> integrate Gateway code into the Flink codebase.
> 
> 
> In the future, we can add a feature that merges SQL gateway into
> JobManager. We can request JobManager API to directly submit the Flink SQL
> job. It will further improve the performance of Flink OLAP.  In the future,
> the Flink must be a unified engine for batch, stream, and OLAP. The
> Presto/Trino directly requests the master node to submit a job, if so, we
> can reduce Q&M in Flink session mode. Perhaps, the Flink application mode
> can’t merge SQL gateway into JobManager, but Flink OLAP almost always uses
> session mode.
> 
>> Gateway to support multiple Flink versions
> 
> 
> If we will merge the SQL gateway into JobManager, the SQL Gateway itself
> can adapt only one Flink version. We could import a Network Gateway to
> redirect requests to Gateway or JobManager of various versions. Perhaps,
> the network gateway uses other projects, like Apache Kyuubi or Zeppelin,
> etc.
> 
>> I don't think that the Gateway is a 'core' function of Flink which should
> 
> be included with Flink.
> 
> In the production environment, Flink SQL always uses a Gateway. This point
> can be observed in the user email lists and some Flink Forward topics. The
> SQL Gateway is an important infrastructure for big data compute engine. As
> the Flink has not it, many Flink users achieve SQL Gateway in the Apache
> Kyuubi project, but it should be the work of official Flink.
> 
>> I think it's fine to move this functionlity to the client rather than
> 
> gateway. WDYT?
> 
> I agree with the `init-file` option in the client. I think the `init-file`
> functionality in Gateway is NOT important in the first version of Gateway.
> Now, the hive JDBC option ‘initFile’ already has this functionality. After
> SQL Gateway releases and we observe feedback from the community, we maybe
> will discuss this problem again.
> 
> Best,
> 
> LuNing Wang.
> 
> 
> Shengkai Fang <fs...@gmail.com> 于2022年5月6日周五 14:34写道：
> 
>> Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback
>> 
>>> I would like to understand why it's complicated to make the upgrades
>>> problematic
>> 
>> I aggree with Jark's point. The API is not very stable in the Flink
>> actually. For example, the Gateway relies on the planner. But in
>> release-1.14 Flink renames the blink planner package. In release-1.15 Flink
>> makes the planner scala free, which means other projects should not
>> directly rely on the planner.
>> 
>>> Does the Flink SQL gateway support submitting a batch job?
>> 
>> Of course. In the SQL Gateway, you can just use the sql SET
>> 'execution.runtime-mode' = 'batch' to switch to the batch environment. Then
>> the job you submit later will be executed in the batch mode.
>> 
>>> The architecture of the Gateway is in the following graph.
>> Is the TableEnvironment shared for all sessions ?
>> 
>> No. Every session has its individual TableEnvironment. I have modified the
>> graph to make everything more clear.
>> 
>>> /v1/sessions
>>>> Are both local file and remote file supported for `libs` and `jars`?
>> 
>> We don't limit the usage here. But I think we will only support the local
>> file in the next version.
>> 
>>>> Does sql gateway support upload files?
>> 
>> No. We need a new API to do this. We can collect more user feedback to
>> determine whether we need to implement this feature.
>> 
>>> /v1/sessions/:session_handle/configure_session
>>>> Can this api be replaced with `/v1/sessions/:session_handle/statements`
>> ?
>> 
>> Actually the API above is different. The
>> `/v1/sessions/:session_handle/configure_session` API uses SQL to configure
>> the environment, which only allows the limited types of SQL. But the
>> `/v1/sessions/:session_handle/statements` has no limitation. I think we'd
>> better use a different API to distinguish these.
>> 
>>> /v1/sessions/:session_id/operations/:operation_handle/status
>>>> `:session_id` is a typo, it should be `:session_handdle`
>> 
>> Yes. I have fixed the mistake.
>> 
>>> /v1/sessions/:session_handle/statements
>>> The statement must be a single command
>> 
>>>> Does this api support `begin statement set ... end` or `statement set
>>>> begin ... end`?
>> 
>> For BEGIN STATEMENT SET, it will open a buffer in the Session and allows
>> the users to submit the insert statement into the Session later. When the
>> Session receives the END statement, the Gateway will submit the buffered
>> statements.
>> 
>> For STATEMENT SET BEGIN ... END, the parser is able to parse the statement.
>> We can treat it as other SQL.
>> 
>>>> DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
>> 
>> For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
>> just add it into the class path or remove it from the class path. If the
>> jar is the remote jar, we will create a session level directory and
>> download the jar into the directory. When the session closes, it should
>> also clean up all the resources in the session-level directory.
>> 
>> I have updated the FLIP to add more info about these.
>> 
>>> /v1/sessions/:session_handle/operations/:operation_handle/result/:token
>>> "type": # string value of LogicalType
>>>> Some LogicalTypes can not be serialized, such as: CharType(0)
>> 
>> I think it's about the serialization of the LogicalType. We can follow the
>> behaviour in the LogicalTypeJsonSerializer.
>> 
>>> endpoint.protocol
>>>> I think REST is not a kind of protocol[1], but is an architectural style.
>>>> The value should be `HTTP`.
>> 
>> I still prefer to use the REST as the value because REST also allows the
>> HTTPS as the protocol. After offline discussion with Godfrey, we think it's
>> better to use the 'endpoint.type' instead.
>> 
>>> Catalog API
>>> ...
>>>> I think we should avoid providing such api, because once catalog api
>>>> is changed or added,
>>>> This class should also be changed. SQL statement is a more general
>> interface.
>> 
>> The exposed API is used by the endpoint to organize its required output.
>> The main problem in your plan is that it requires us to parse the data from
>> the RowData, which only contains the basic types. I think it's much more
>> difficult to maintain compared to the current plan that returns structured
>> objects. I think the GatewayService relies on the Catalog but it doesn't
>> mean the GatewayService should expose all the API exposed by the Catalog.
>> 
>>> Options
>>>> sql-gateway.session.idle.timeout
>>>> sql-gateway.session.check.interval
>>>> sql-gateway.worker.keepalive.time
>> 
>> Okay. I have updated the FLIP about the option names.
>> 
>>> why do we need both the rest api and the SQLGatewayService
>>> API, maybe I'm missing something, what's the difference between them?
>> 
>> REST API is the user interface. The REST API transforms the request to the
>> invocation of the SQLGatewayService that is the one doing the work. We
>> split the Gateway into the SQLGatewayService and Endpoint(REST API) and its
>> benefit is that all the Endpoints share the same SQLGatewayService.
>> 
>>> Is it possible to use one set of rest api to solve all the problems?
>> 
>> I think we can't. I don't understand the meaning of all the problems. We
>> can use the REST API to expose all the functionalities in the Gateway side.
>> But many users may have their tools to communicate to the Gateway, which
>> may be based on the HiveServer2 API(thrift api).
>> 
>> Best,
>> Shengkai
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Jingsong Li <ji...@gmail.com> 于2022年5月6日周五 09:16写道：
>> 
>>> Thanks Shengkai for driving.  And all for your discussion.
>>> 
>>>> The reason why we introduce the gateway with pluggable endpoints is
>> that
>>> many users has their preferences. For example, the HiveServer2 users
>> prefer
>>> to use the gateway with HiveServer2-style API, which has numerous tools.
>>> However, some filnk-native users may prefer to use the REST API.
>> Therefore,
>>> we hope to learn from the Kyuubi's design that expose multiple endpoints
>>> with different API that allow the user to use.
>>> 
>>> My understanding is that we need multiple endpoints, But I don't quite
>>> understand why we need both the rest api and the SQLGatewayService
>>> API, maybe I'm missing something, what's the difference between them?
>>> Is it possible to use one set of rest api to solve all the problems?
>>> 
>>>> Gateway to support multiple Flink versions
>>> 
>>> I think this is a good question to consider.
>>> - First of all, I think it is absolutely impossible for gateway to
>>> support multiple versions of Flink under the current architecture,
>>> because gateway relies on Flink SQL and a lot of SQL compiled and
>>> optimized code is bound to the Flink version.
>>> - The other way is that gateway does not rely on Flink SQL, and each
>>> time a different version of Flink Jar is loaded to compile the job at
>>> once, and frankly speaking, stream jobs actually prefer this model.
>>> 
>>> The benefit of gateway support for multiple versions is that it's
>>> really more user-friendly. I've seen cases where users must have
>>> multiple versions existing in a cluster, and if each version needs to
>>> run a gateway, the O&M burden will be heavy.
>>> 
>>>> I don't think that the Gateway is a 'core' function of Flink which
>>> should be included with Flink.
>>> 
>>> First, I think the Gateway is a 'core' function in Flink.
>>> Why?
>>> I think our architecture should be consistent, which means that Flink
>>> sql-client should use the implementation of gateway, which means that
>>> sql-client depends on gateway.
>>> And sql-client is the basic tool of flink sql, it must exist in flink
>>> repository, otherwise flink sql has no most important entrance.
>>> So, the gateway itself should be our core functionality as well.
>>> 
>>> Best,
>>> Jingsong
>>> 
>>>> On Thu, May 5, 2022 at 10:06 PM Jark Wu <im...@gmail.com> wrote:
>>>>> 
>>>>> Hi Martijn,
>>>>> 
>>>>> Regarding maintaining Gateway inside or outside Flink code base,
>>>>> I would like to share my thoughts:
>>>>> 
>>>>>> I would like to understand why it's complicated to make the upgrades
>>>>> problematic. Is it because of relying on internal interfaces? If so,
>>> should
>>>> we not consider making them public?
>>>> 
>>>> It's not about internal interfaces. Flink itself doesn't provide
>> backward
>>>> compatibility for public APIs.
>>>> 
>>>> 
>>>>> a) it will not be possible to have separate releases of the Gateway,
>>>> they will be tied to individual Flink releases
>>>> I don't think it's a problem. On the contrary, maintaining a separate
>>> repo
>>>> for Gateway will take a lot of
>>>> extra community efforts, e.g., individual CICD, docs, releases.
>>>> 
>>>> 
>>>>> b) if you want the Gateway to support multiple Flink versions
>>>> Sorry, I don't see any users requesting this feature for such a long
>> time
>>>> for SQL Gateway.
>>>> Users can build services on Gateway to easily support multi Flink
>>> versions
>>>> (a Gateway for a Flink version).
>>>> It's difficult for Gateway to support multi-version because Flink
>> doesn't
>>>> provide an API that supports backward and forward compatibility.
>>>> If Gateway wants to support multi-version, it has to invent an
>>>> inner-gateway for each version, and Gateway act as a proxy to
>> communicate
>>>> with inner-gateway.
>>>> So you have to have a gateway to couple with the Flink version.
>>>> 
>>>> In fact, Gateway is the layer to support multi Flink versions for
>>>> higher-level applications because its API (REST, gRpc) provides
>> backward
>>>> and forward compatibility.
>>>> The gateway itself doesn't need to support multi Flink versions.
>> Besides,
>>>> Trino/Presto also provides servers[1] for each version.
>>>> 
>>>> 
>>>>> I don't think that the Gateway is a 'core' function of Flink which
>>> should
>>>> be included with Flink.
>>>> Sorry, I can't agree with this. If I remember correctly, Flink SQL has
>>> been
>>>> promoted to first-class citizen for a long time.
>>>> The community also aims to make Flink a truly batch-stream unified
>>>> computing platform, and Gateway would be the entry and center of the
>>>> platform.
>>>> From my point of view, Gateway is a very "core" function and must be
>>>> included in Flink to have better cooperation with SQL and provide an
>>>> out-of-box experience.
>>>> 
>>>> Best,
>>>> Jark
>>>> 
>>>> [1]: https://trino.io/download.html
>>>> 
>>>> On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com> wrote:
>>>> 
>>>>> Hi Shengkai.
>>>>> 
>>>>> Thanks for driving the proposal, it's been silent too long.
>>>>> 
>>>>> I have a few questions:
>>>>> about the Architecture
>>>>>> The architecture of the Gateway is in the following graph.
>>>>> Is the TableEnvironment shared for all sessions ?
>>>>> 
>>>>> about the REST Endpoint
>>>>>> /v1/sessions
>>>>> Are both local file and remote file supported for `libs` and `jars`?
>>>>> Does sql gateway support upload file?
>>>>> 
>>>>>> /v1/sessions/:session_handle/configure_session
>>>>> Can this api be replaced with
>>> `/v1/sessions/:session_handle/statements` ?
>>>>> 
>>>>>> /v1/sessions/:session_id/operations/:operation_handle/status
>>>>> `:session_id` is a typo, it should be `:session_handdle`
>>>>> 
>>>>>> /v1/sessions/:session_handle/statements
>>>>>> The statement must be a single command
>>>>> Does this api support `begin statement set ... end` or `statement set
>>>>> begin ... end`
>>>>> DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
>>>>> 
>>>>> 
>>>> /v1/sessions/:session_handle/operations/:operation_handle/result/:token
>>>>>> "type": # string value of LogicalType
>>>>> Some LogicalTypes can not be serialized, such as: CharType(0)
>>>>> 
>>>>> about Options
>>>>>> endpoint.protocol
>>>>> I think REST is not a kind of protocol[1], but is an architectural
>>> style.
>>>>> The value should be `HTTP`.
>>>>> 
>>>>> about SQLGatewayService API
>>>>>> Catalog API
>>>>>> ...
>>>>> I think we should avoid providing such api, because once catalog api
>>>>> is changed or added,
>>>>> This class should also be changed. SQL statement is a more general
>>>>> interface.
>>>>> 
>>>>>> Options
>>>>>> sql-gateway.session.idle.timeout
>>>>>> sql-gateway.session.check.interval
>>>>>> sql-gateway.worker.keepalive.time
>>>>> It's better we can keep the option style as Flink, the level should
>>>>> not be too deep.
>>>>> sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
>>>>> sql-gateway.session.check.interval ->
>>> sql-gateway.session.check-interval
>>>>> sql-gateway.worker.keepalive.time ->
>> sql-gateway.worker.keepalive->time
>>>>> 
>>>>> [1] https://restfulapi.net/
>>>>> 
>>>>> Best,
>>>>> Godfrey
>>>>> 
>>>>> Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
>>>>>> 
>>>>>> Hi Shengkai,
>>>>>> 
>>>>>> I have another concern about the submission of batch job. Does the
>>> Flink
>>>>> SQL gateway support to submit batch job? In Kyuubi,
>>> BatchProcessBuilder is
>>>>> used to submit batch job. What about the Flink SQL gateway?
>>>>>> 
>>>>>> Best regards,
>>>>>> Nicholas Jiang
>>>>>> 
>>>>>> On 2022/04/24 03:28:36 Shengkai Fang wrote:
>>>>>>> Hi. Jiang.
>>>>>>> 
>>>>>>> Thanks for your feedback！
>>>>>>> 
>>>>>>>> Do the public interfaces of GatewayService refer to any
>> service?
>>>>>>> 
>>>>>>> We will only expose one GatewayService implementation. We will
>> put
>>> the
>>>>>>> interface into the common package and the developer who wants to
>>>>> implement
>>>>>>> a new endpoint can just rely on the interface package rather than
>>> the
>>>>>>> implementation.
>>>>>>> 
>>>>>>>> What's the behavior of SQL Client Gateway working on Yarn or
>> K8S?
>>>>> Does
>>>>>>> the SQL Client Gateway support application or session mode on
>> Yarn?
>>>>>>> 
>>>>>>> I think we can support SQL Client Gateway to submit the jobs in
>>>>>>> application/sesison mode.
>>>>>>> 
>>>>>>>> Is there any event trigger in the operation state machine?
>>>>>>> 
>>>>>>> Yes. I have already updated the content and add more details
>> about
>>> the
>>>>>>> state machine. During the revise, I found that I mix up the two
>>>>> concepts:
>>>>>>> job submission and job execution. In fact, we only control the
>>>>> submission
>>>>>>> mode at the gateway layer. Therefore, we don't need to mapping
>> the
>>>>>>> JobStatus here. If the user expects that the synchronization
>>> behavior
>>>>> is to
>>>>>>> wait for the completion of the job execution before allowing the
>>> next
>>>>>>> statement to be executed, then the Operation lifecycle should
>> also
>>>>> contains
>>>>>>> the job's execution, which means users should set
>> `table.dml-sync`.
>>>>>>> 
>>>>>>>> What's the return schema for the public interfaces of
>>> GatewayService?
>>>>>>> Like getTable interface, what's the return value schema?
>>>>>>> 
>>>>>>> The API of the GatewayService return the java objects and the
>>> endpoint
>>>>> can
>>>>>>> organize the objects with expected schema. The return results is
>>> also
>>>>> list
>>>>>>> the section ComponetAPI#GatewayService#API. The return type of
>> the
>>>>>>> GatewayService#getTable is `ContextResolvedTable`.
>>>>>>> 
>>>>>>>> How does the user get the operation log?
>>>>>>> 
>>>>>>> The OperationManager will register the LogAppender before the
>>> Operation
>>>>>>> execution. The Log Appender will hijack the logger and also write
>>> the
>>>>> log
>>>>>>> that related to the Operation to another files. When users wants
>> to
>>>>> fetch
>>>>>>> the Operation log, the GatewayService will read the content in
>> the
>>>>> file and
>>>>>>> return.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Shengkai
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
>>>>>>> 
>>>>>>>> Hi Shengkai.
>>>>>>>> 
>>>>>>>> Thanks for driving the proposal of SQL Client Gateway. I have
>>> some
>>>>>>>> knowledge of Kyuubi and have some questions about the design:
>>>>>>>> 
>>>>>>>> 1.Do the public interfaces of GatewayService refer to any
>>> service? If
>>>>>>>> referring to HiveService, does GatewayService need interfaces
>>> like
>>>>>>>> getQueryId etc.
>>>>>>>> 
>>>>>>>> 2.What's the behavior of SQL Client Gateway working on Yarn or
>>> K8S?
>>>>> Does
>>>>>>>> the SQL Client Gateway support application or session mode on
>>> Yarn?
>>>>>>>> 
>>>>>>>> 3.Is there any event trigger in the operation state machine?
>>>>>>>> 
>>>>>>>> 4.What's the return schema for the public interfaces of
>>>>> GatewayService?
>>>>>>>> Like getTable interface, what's the return value schema?
>>>>>>>> 
>>>>>>>> 5.How does the user get the operation log?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Nicholas Jiang
>>>>>>>> 
>>>>>>>> On 2022/04/21 06:42:30 Shengkai Fang wrote:
>>>>>>>>> Hi, Flink developers.
>>>>>>>>> 
>>>>>>>>> I want to start a discussion about the FLIP-91: Support Flink
>>> SQL
>>>>>>>>> Gateway[1]. Flink SQL Gateway is a service that allows users
>> to
>>>>> submit
>>>>>>>> and
>>>>>>>>> manage their jobs in the online environment with the
>> pluggable
>>>>> endpoints.
>>>>>>>>> The reason why we introduce the Gateway with pluggable
>>> endpoints
>>>>> is that
>>>>>>>>> many users have their preferences. For example, the
>> HiveServer2
>>>>> users
>>>>>>>>> prefer to use the gateway with HiveServer2-style API, which
>> has
>>>>> numerous
>>>>>>>>> tools. However, some filnk-native users may prefer to use the
>>> REST
>>>>> API.
>>>>>>>>> Therefore, we propose the SQL Gateway with pluggable
>> endpoint.
>>>>>>>>> 
>>>>>>>>> In the FLIP, we also propose the REST endpoint, which has the
>>>>> similar
>>>>>>>>> APIs compared to the gateway in the
>>>>> ververica/flink-sql-gateway[2]. At
>>>>>>>> the
>>>>>>>>> last, we discuss how to use the SQL Client to submit the
>>> statement
>>>>> to the
>>>>>>>>> Gateway with the REST API.
>>>>>>>>> 
>>>>>>>>> I am glad that you can give some feedback about FLIP-91.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> Shengkai
>>>>>>>>> 
>>>>>>>>> [1]
>>>>>>>>> 
>>>>>>>> 
>>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
>>>>>>>>> [2] https://github.com/ververica/flink-sql-gateway
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Martijn Visser <ma...@ververica.com>]

Hi everyone,

Happy to see that this discussion is very much active. Couple of comments:

> It's not about internal interfaces. Flink itself doesn't provide backward
compatibility for public APIs.

Is that so? In FLIP-196 [1] is explicitly stated "What we guarantee in
terms of stability is that a program written against a public API will
compile w/o errors when upgrading Flink (API backwards compatibility)".

> Sorry, I don't see any users requesting this feature for such a long time
for SQL Gateway.
> So you have to have a gateway to couple with the Flink version.

I would be interested to hear the opinion of users on this. I have no
strong opinion on this, I could see value in having multiple Flink version
support in a Gateway but if there's no user demand for it, then fine. I
could imagine that multiple versions support is desired by end-users, but
that it hasn't been requested yet because of the current state of the SQL
gateway. We can also simply place it out of scope for now and say that
multiple version support could be realised by another component, but with
an external component that uses the Gateway from the individual Flink
releases.

On the topic of having the Gateway in the Flink repo itself, I liked
Jingsong's argument that the SQL Client should use the Gateway, which
creates a dependency. Given that I do think the SQL Client is an important
starter functionality, that does give a compelling argument to include the
Gateway in the Flink repository. I could agree with that, as long as we do
commit that the Gateway uses properly defined public interfaces. So this
FLIP should follow FLIP-196 [1] and and FLIP-197 [2]

> In the future, the Flink must be a unified engine for batch, stream, and
OLAP.

While I understand and know the focus of some maintainers on OLAP, I do
think that the Flink project has not made any official decision that it
**must** include OLAP. The current scope is still a unified engine for
batch and streaming. As long as OLAP improvements don't hurt or cause
problems with the unified batch and streaming, there's no issue of course.
But I am careful because we know there can and will be situations where
OLAP features could conflict with unified batch and streaming (and vice
versa of course).

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-196%3A+Source+API+stability+guarantees
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process


On Fri, 6 May 2022 at 12:05, godfrey he <go...@gmail.com> wrote:

> Hi Martijn, Shengkai
>
> >I don't think that the Gateway is a 'core'
> >function of Flink which should be included with Flink.
> I have a different viewpoint. For SQL users, it provides out-of-box
> experience.
> Just like SQL-client, users can enjoy variable sql experiences after
> downloading/building a Flink package. In the design, SQL-client is
> just a client/part of SQL-gateway.
> We can find that, for HiveServer2 and Presto, the similar role is also
> built-in.
>
> > the Gateway to support multiple Flink versions
> I think this idea is good for users, who can use one service to
> support multiple Flink versions. But I do not think current design
> should support it.
> As we know, the API has changed a lot in recent versions of Flink,
> and it's so hard to put different versions of Flink's code into one process
>  without any class conflicts.
> The easiest way to support it would be to use a sub-process model.
> For each sub-process, it's the sql gateway we are discussing now.
> We can start another project (it can be outside of the Flink project)
> to support it.
>
> > For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
> >just add it into the class path or remove it from the class path.
> The client and the Gateway service may be on different machines.
>
> Best,
> Godfrey
>
> LuNing Wang <wa...@gmail.com> 于2022年5月6日周五 17:06写道：
> >
> > Thanks, Shengkai for driving.  And all for your discussion.
> >
> >
> >
> > > intergate the Gateway into the Flink code base
> >
> > After I talk with Shengkai offline and read the topic `Powering HTAP at
> > ByteDance with Apache Flink` of Flink Forward Asia. I think it is better
> to
> > integrate Gateway code into the Flink codebase.
> >
> >
> > In the future, we can add a feature that merges SQL gateway into
> > JobManager. We can request JobManager API to directly submit the Flink
> SQL
> > job. It will further improve the performance of Flink OLAP.  In the
> future,
> > the Flink must be a unified engine for batch, stream, and OLAP. The
> > Presto/Trino directly requests the master node to submit a job, if so, we
> > can reduce Q&M in Flink session mode. Perhaps, the Flink application mode
> > can’t merge SQL gateway into JobManager, but Flink OLAP almost always
> uses
> > session mode.
> >
> > > Gateway to support multiple Flink versions
> >
> >
> > If we will merge the SQL gateway into JobManager, the SQL Gateway itself
> > can adapt only one Flink version. We could import a Network Gateway to
> > redirect requests to Gateway or JobManager of various versions. Perhaps,
> > the network gateway uses other projects, like Apache Kyuubi or Zeppelin,
> > etc.
> >
> > > I don't think that the Gateway is a 'core' function of Flink which
> should
> >
> > be included with Flink.
> >
> > In the production environment, Flink SQL always uses a Gateway. This
> point
> > can be observed in the user email lists and some Flink Forward topics.
> The
> > SQL Gateway is an important infrastructure for big data compute engine.
> As
> > the Flink has not it, many Flink users achieve SQL Gateway in the Apache
> > Kyuubi project, but it should be the work of official Flink.
> >
> > > I think it's fine to move this functionlity to the client rather than
> >
> > gateway. WDYT?
> >
> > I agree with the `init-file` option in the client. I think the
> `init-file`
> > functionality in Gateway is NOT important in the first version of
> Gateway.
> > Now, the hive JDBC option ‘initFile’ already has this functionality.
> After
> > SQL Gateway releases and we observe feedback from the community, we maybe
> > will discuss this problem again.
> >
> > Best,
> >
> > LuNing Wang.
> >
> >
> > Shengkai Fang <fs...@gmail.com> 于2022年5月6日周五 14:34写道：
> >
> > > Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback
> > >
> > > > I would like to understand why it's complicated to make the upgrades
> > > > problematic
> > >
> > > I aggree with Jark's point. The API is not very stable in the Flink
> > > actually. For example, the Gateway relies on the planner. But in
> > > release-1.14 Flink renames the blink planner package. In release-1.15
> Flink
> > > makes the planner scala free, which means other projects should not
> > > directly rely on the planner.
> > >
> > > >  Does the Flink SQL gateway support submitting a batch job?
> > >
> > > Of course. In the SQL Gateway, you can just use the sql SET
> > > 'execution.runtime-mode' = 'batch' to switch to the batch environment.
> Then
> > > the job you submit later will be executed in the batch mode.
> > >
> > > > The architecture of the Gateway is in the following graph.
> > > Is the TableEnvironment shared for all sessions ?
> > >
> > > No. Every session has its individual TableEnvironment. I have modified
> the
> > > graph to make everything more clear.
> > >
> > > > /v1/sessions
> > > >> Are both local file and remote file supported for `libs` and `jars`?
> > >
> > > We don't limit the usage here. But I think we will only support the
> local
> > > file in the next version.
> > >
> > > >> Does sql gateway support upload files?
> > >
> > > No. We need a new API to do this. We can collect more user feedback to
> > > determine whether we need to implement this feature.
> > >
> > > >/v1/sessions/:session_handle/configure_session
> > > >> Can this api be replaced with
> `/v1/sessions/:session_handle/statements`
> > > ?
> > >
> > > Actually the API above is different. The
> > > `/v1/sessions/:session_handle/configure_session` API uses SQL to
> configure
> > > the environment, which only allows the limited types of SQL. But the
> > > `/v1/sessions/:session_handle/statements` has no limitation. I think
> we'd
> > > better use a different API to distinguish these.
> > >
> > > >/v1/sessions/:session_id/operations/:operation_handle/status
> > > >>`:session_id` is a typo, it should be `:session_handdle`
> > >
> > > Yes. I have fixed the mistake.
> > >
> > > >/v1/sessions/:session_handle/statements
> > > >The statement must be a single command
> > >
> > > >> Does this api support `begin statement set ... end` or `statement
> set
> > > >> begin ... end`?
> > >
> > > For BEGIN STATEMENT SET, it will open a buffer in the Session and
> allows
> > > the users to submit the insert statement into the Session later. When
> the
> > > Session receives the END statement, the Gateway will submit the
> buffered
> > > statements.
> > >
> > > For STATEMENT SET BEGIN ... END, the parser is able to parse the
> statement.
> > > We can treat it as other SQL.
> > >
> > > >> DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
> > >
> > > For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
> > > just add it into the class path or remove it from the class path. If
> the
> > > jar is the remote jar, we will create a session level directory and
> > > download the jar into the directory. When the session closes, it should
> > > also clean up all the resources in the session-level directory.
> > >
> > > I have updated the FLIP to add more info about these.
> > >
> > >
> >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > > >"type": # string value of LogicalType
> > > >> Some LogicalTypes can not be serialized, such as: CharType(0)
> > >
> > > I think it's about the serialization of the LogicalType. We can follow
> the
> > > behaviour in the LogicalTypeJsonSerializer.
> > >
> > > > endpoint.protocol
> > > >>I think REST is not a kind of protocol[1], but is an architectural
> style.
> > > >> The value should be `HTTP`.
> > >
> > > I still prefer to use the REST as the value because REST also allows
> the
> > > HTTPS as the protocol. After offline discussion with Godfrey, we think
> it's
> > > better to use the 'endpoint.type' instead.
> > >
> > > >  Catalog API
> > > > ...
> > > >> I think we should avoid providing such api, because once catalog api
> > > >> is changed or added,
> > > >> This class should also be changed. SQL statement is a more general
> > > interface.
> > >
> > > The exposed API is used by the endpoint to organize its required
> output.
> > > The main problem in your plan is that it requires us to parse the data
> from
> > > the RowData, which only contains the basic types. I think it's much
> more
> > > difficult to maintain compared to the current plan that returns
> structured
> > > objects. I think the GatewayService relies on the Catalog but it
> doesn't
> > > mean the GatewayService should expose all the API exposed by the
> Catalog.
> > >
> > > > Options
> > > >> sql-gateway.session.idle.timeout
> > > >> sql-gateway.session.check.interval
> > > >> sql-gateway.worker.keepalive.time
> > >
> > > Okay. I have updated the FLIP about the option names.
> > >
> > > >why do we need both the rest api and the SQLGatewayService
> > > >API, maybe I'm missing something, what's the difference between them?
> > >
> > > REST API is the user interface. The REST API transforms the request to
> the
> > > invocation of the SQLGatewayService that is the one doing the work. We
> > > split the Gateway into the SQLGatewayService and Endpoint(REST API)
> and its
> > > benefit is that all the Endpoints share the same SQLGatewayService.
> > >
> > > > Is it possible to use one set of rest api to solve all the problems?
> > >
> > > I think we can't. I don't understand the meaning of all the problems.
> We
> > > can use the REST API to expose all the functionalities in the Gateway
> side.
> > > But many users may have their tools to communicate to the Gateway,
> which
> > > may be based on the HiveServer2 API(thrift api).
> > >
> > > Best,
> > > Shengkai
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Jingsong Li <ji...@gmail.com> 于2022年5月6日周五 09:16写道：
> > >
> > > > Thanks Shengkai for driving.  And all for your discussion.
> > > >
> > > > > The reason why we introduce the gateway with pluggable endpoints is
> > > that
> > > > many users has their preferences. For example, the HiveServer2 users
> > > prefer
> > > > to use the gateway with HiveServer2-style API, which has numerous
> tools.
> > > > However, some filnk-native users may prefer to use the REST API.
> > > Therefore,
> > > > we hope to learn from the Kyuubi's design that expose multiple
> endpoints
> > > > with different API that allow the user to use.
> > > >
> > > > My understanding is that we need multiple endpoints, But I don't
> quite
> > > > understand why we need both the rest api and the SQLGatewayService
> > > > API, maybe I'm missing something, what's the difference between them?
> > > > Is it possible to use one set of rest api to solve all the problems?
> > > >
> > > > > Gateway to support multiple Flink versions
> > > >
> > > > I think this is a good question to consider.
> > > > - First of all, I think it is absolutely impossible for gateway to
> > > > support multiple versions of Flink under the current architecture,
> > > > because gateway relies on Flink SQL and a lot of SQL compiled and
> > > > optimized code is bound to the Flink version.
> > > > - The other way is that gateway does not rely on Flink SQL, and each
> > > > time a different version of Flink Jar is loaded to compile the job at
> > > > once, and frankly speaking, stream jobs actually prefer this model.
> > > >
> > > > The benefit of gateway support for multiple versions is that it's
> > > > really more user-friendly. I've seen cases where users must have
> > > > multiple versions existing in a cluster, and if each version needs to
> > > > run a gateway, the O&M burden will be heavy.
> > > >
> > > > > I don't think that the Gateway is a 'core' function of Flink which
> > > > should be included with Flink.
> > > >
> > > > First, I think the Gateway is a 'core' function in Flink.
> > > > Why?
> > > > I think our architecture should be consistent, which means that Flink
> > > > sql-client should use the implementation of gateway, which means that
> > > > sql-client depends on gateway.
> > > > And sql-client is the basic tool of flink sql, it must exist in flink
> > > > repository, otherwise flink sql has no most important entrance.
> > > > So, the gateway itself should be our core functionality as well.
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Thu, May 5, 2022 at 10:06 PM Jark Wu <im...@gmail.com> wrote:
> > > > >
> > > > > Hi Martijn,
> > > > >
> > > > > Regarding maintaining Gateway inside or outside Flink code base,
> > > > > I would like to share my thoughts:
> > > > >
> > > > > > I would like to understand why it's complicated to make the
> upgrades
> > > > > problematic. Is it because of relying on internal interfaces? If
> so,
> > > > should
> > > > > we not consider making them public?
> > > > >
> > > > > It's not about internal interfaces. Flink itself doesn't provide
> > > backward
> > > > > compatibility for public APIs.
> > > > >
> > > > >
> > > > > > a) it will not be possible to have separate releases of the
> Gateway,
> > > > > they will be tied to individual Flink releases
> > > > > I don't think it's a problem. On the contrary, maintaining a
> separate
> > > > repo
> > > > > for Gateway will take a lot of
> > > > > extra community efforts, e.g., individual CICD, docs, releases.
> > > > >
> > > > >
> > > > > > b) if you want the Gateway to support multiple Flink versions
> > > > > Sorry, I don't see any users requesting this feature for such a
> long
> > > time
> > > > > for SQL Gateway.
> > > > > Users can build services on Gateway to easily support multi Flink
> > > > versions
> > > > > (a Gateway for a Flink version).
> > > > > It's difficult for Gateway to support multi-version because Flink
> > > doesn't
> > > > > provide an API that supports backward and forward compatibility.
> > > > > If Gateway wants to support multi-version, it has to invent an
> > > > > inner-gateway for each version, and Gateway act as a proxy to
> > > communicate
> > > > > with inner-gateway.
> > > > > So you have to have a gateway to couple with the Flink version.
> > > > >
> > > > > In fact, Gateway is the layer to support multi Flink versions for
> > > > > higher-level applications because its API (REST, gRpc) provides
> > > backward
> > > > > and forward compatibility.
> > > > > The gateway itself doesn't need to support multi Flink versions.
> > > Besides,
> > > > > Trino/Presto also provides servers[1] for each version.
> > > > >
> > > > >
> > > > > > I don't think that the Gateway is a 'core' function of Flink
> which
> > > > should
> > > > > be included with Flink.
> > > > > Sorry, I can't agree with this. If I remember correctly, Flink SQL
> has
> > > > been
> > > > > promoted to first-class citizen for a long time.
> > > > > The community also aims to make Flink a truly batch-stream unified
> > > > > computing platform, and Gateway would be the entry and center of
> the
> > > > > platform.
> > > > > From my point of view, Gateway is a very "core" function and must
> be
> > > > > included in Flink to have better cooperation with SQL and provide
> an
> > > > > out-of-box experience.
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > [1]: https://trino.io/download.html
> > > > >
> > > > > On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Shengkai.
> > > > > >
> > > > > > Thanks for driving the proposal, it's been silent too long.
> > > > > >
> > > > > > I have a few questions:
> > > > > > about the Architecture
> > > > > > > The architecture of the Gateway is in the following graph.
> > > > > > Is the TableEnvironment shared for all sessions ?
> > > > > >
> > > > > > about the REST Endpoint
> > > > > > > /v1/sessions
> > > > > > Are both local file and remote file supported for `libs` and
> `jars`?
> > > > > > Does sql gateway support upload file?
> > > > > >
> > > > > > >/v1/sessions/:session_handle/configure_session
> > > > > > Can this api be replaced with
> > > > `/v1/sessions/:session_handle/statements` ?
> > > > > >
> > > > > > >/v1/sessions/:session_id/operations/:operation_handle/status
> > > > > > `:session_id` is a typo, it should be `:session_handdle`
> > > > > >
> > > > > > >/v1/sessions/:session_handle/statements
> > > > > > >The statement must be a single command
> > > > > > Does this api support `begin statement set ... end` or
> `statement set
> > > > > > begin ... end`
> > > > > >  DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the
> jars?
> > > > > >
> > > > > >
> > > >
> >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > > > > > >"type": # string value of LogicalType
> > > > > >  Some LogicalTypes can not be serialized, such as: CharType(0)
> > > > > >
> > > > > > about Options
> > > > > > > endpoint.protocol
> > > > > > I think REST is not a kind of protocol[1], but is an
> architectural
> > > > style.
> > > > > > The value should be `HTTP`.
> > > > > >
> > > > > > about SQLGatewayService API
> > > > > > >  Catalog API
> > > > > > > ...
> > > > > > I think we should avoid providing such api, because once catalog
> api
> > > > > > is changed or added,
> > > > > > This class should also be changed. SQL statement is a more
> general
> > > > > > interface.
> > > > > >
> > > > > > > Options
> > > > > > > sql-gateway.session.idle.timeout
> > > > > > >sql-gateway.session.check.interval
> > > > > > >sql-gateway.worker.keepalive.time
> > > > > > It's better we can keep the option style as Flink, the level
> should
> > > > > > not be too deep.
> > > > > > sql-gateway.session.idle.timeout ->
> sql-gateway.session.idle-timeout
> > > > > > sql-gateway.session.check.interval ->
> > > > sql-gateway.session.check-interval
> > > > > > sql-gateway.worker.keepalive.time ->
> > > sql-gateway.worker.keepalive->time
> > > > > >
> > > > > > [1] https://restfulapi.net/
> > > > > >
> > > > > > Best,
> > > > > > Godfrey
> > > > > >
> > > > > > Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
> > > > > > >
> > > > > > > Hi Shengkai,
> > > > > > >
> > > > > > > I have another concern about the submission of batch job. Does
> the
> > > > Flink
> > > > > > SQL gateway support to submit batch job? In Kyuubi,
> > > > BatchProcessBuilder is
> > > > > > used to submit batch job. What about the Flink SQL gateway?
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Nicholas Jiang
> > > > > > >
> > > > > > > On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > > > > > > > Hi. Jiang.
> > > > > > > >
> > > > > > > > Thanks for your feedback！
> > > > > > > >
> > > > > > > > > Do the public interfaces of GatewayService refer to any
> > > service?
> > > > > > > >
> > > > > > > > We will only expose one GatewayService implementation. We
> will
> > > put
> > > > the
> > > > > > > > interface into the common package and the developer who
> wants to
> > > > > > implement
> > > > > > > > a new endpoint can just rely on the interface package rather
> than
> > > > the
> > > > > > > > implementation.
> > > > > > > >
> > > > > > > > > What's the behavior of SQL Client Gateway working on Yarn
> or
> > > K8S?
> > > > > > Does
> > > > > > > > the SQL Client Gateway support application or session mode on
> > > Yarn?
> > > > > > > >
> > > > > > > > I think we can support SQL Client Gateway to submit the jobs
> in
> > > > > > > > application/sesison mode.
> > > > > > > >
> > > > > > > > > Is there any event trigger in the operation state machine?
> > > > > > > >
> > > > > > > > Yes. I have already updated the content and add more details
> > > about
> > > > the
> > > > > > > > state machine. During the revise, I found that I mix up the
> two
> > > > > > concepts:
> > > > > > > > job submission and job execution. In fact, we only control
> the
> > > > > > submission
> > > > > > > > mode at the gateway layer. Therefore, we don't need to
> mapping
> > > the
> > > > > > > > JobStatus here. If the user expects that the synchronization
> > > > behavior
> > > > > > is to
> > > > > > > > wait for the completion of the job execution before allowing
> the
> > > > next
> > > > > > > > statement to be executed, then the Operation lifecycle should
> > > also
> > > > > > contains
> > > > > > > > the job's execution, which means users should set
> > > `table.dml-sync`.
> > > > > > > >
> > > > > > > > > What's the return schema for the public interfaces of
> > > > GatewayService?
> > > > > > > > Like getTable interface, what's the return value schema?
> > > > > > > >
> > > > > > > > The API of the GatewayService return the java objects and the
> > > > endpoint
> > > > > > can
> > > > > > > > organize the objects with expected schema. The return
> results is
> > > > also
> > > > > > list
> > > > > > > > the section ComponetAPI#GatewayService#API. The return type
> of
> > > the
> > > > > > > > GatewayService#getTable is `ContextResolvedTable`.
> > > > > > > >
> > > > > > > > > How does the user get the operation log?
> > > > > > > >
> > > > > > > > The OperationManager will register the LogAppender before the
> > > > Operation
> > > > > > > > execution. The Log Appender will hijack the logger and also
> write
> > > > the
> > > > > > log
> > > > > > > > that related to the Operation to another files. When users
> wants
> > > to
> > > > > > fetch
> > > > > > > > the Operation log, the GatewayService will read the content
> in
> > > the
> > > > > > file and
> > > > > > > > return.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shengkai
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五
> 16:21写道：
> > > > > > > >
> > > > > > > > > Hi Shengkai.
> > > > > > > > >
> > > > > > > > > Thanks for driving the proposal of SQL Client Gateway. I
> have
> > > > some
> > > > > > > > > knowledge of Kyuubi and have some questions about the
> design:
> > > > > > > > >
> > > > > > > > > 1.Do the public interfaces of GatewayService refer to any
> > > > service? If
> > > > > > > > > referring to HiveService, does GatewayService need
> interfaces
> > > > like
> > > > > > > > > getQueryId etc.
> > > > > > > > >
> > > > > > > > > 2.What's the behavior of SQL Client Gateway working on
> Yarn or
> > > > K8S?
> > > > > > Does
> > > > > > > > > the SQL Client Gateway support application or session mode
> on
> > > > Yarn?
> > > > > > > > >
> > > > > > > > > 3.Is there any event trigger in the operation state
> machine?
> > > > > > > > >
> > > > > > > > > 4.What's the return schema for the public interfaces of
> > > > > > GatewayService?
> > > > > > > > > Like getTable interface, what's the return value schema?
> > > > > > > > >
> > > > > > > > > 5.How does the user get the operation log?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Nicholas Jiang
> > > > > > > > >
> > > > > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > > > > > > > Hi, Flink developers.
> > > > > > > > > >
> > > > > > > > > > I want to start a discussion about the FLIP-91: Support
> Flink
> > > > SQL
> > > > > > > > > > Gateway[1]. Flink SQL Gateway is a service that allows
> users
> > > to
> > > > > > submit
> > > > > > > > > and
> > > > > > > > > > manage their jobs in the online environment with the
> > > pluggable
> > > > > > endpoints.
> > > > > > > > > > The reason why we introduce the Gateway with pluggable
> > > > endpoints
> > > > > > is that
> > > > > > > > > > many users have their preferences. For example, the
> > > HiveServer2
> > > > > > users
> > > > > > > > > > prefer to use the gateway with HiveServer2-style API,
> which
> > > has
> > > > > > numerous
> > > > > > > > > > tools. However, some filnk-native users may prefer to
> use the
> > > > REST
> > > > > > API.
> > > > > > > > > > Therefore, we propose the SQL Gateway with pluggable
> > > endpoint.
> > > > > > > > > >
> > > > > > > > > > In the FLIP, we also propose the REST endpoint, which
> has the
> > > > > > similar
> > > > > > > > > > APIs compared to the gateway in the
> > > > > > ververica/flink-sql-gateway[2]. At
> > > > > > > > > the
> > > > > > > > > > last, we discuss how to use the SQL Client to submit the
> > > > statement
> > > > > > to the
> > > > > > > > > > Gateway with the REST API.
> > > > > > > > > >
> > > > > > > > > > I am glad that you can give some feedback about FLIP-91.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shengkai
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > > > > > > > [2] https://github.com/ververica/flink-sql-gateway
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[godfrey he <go...@gmail.com>]

Hi Martijn, Shengkai

>I don't think that the Gateway is a 'core'
>function of Flink which should be included with Flink.
I have a different viewpoint. For SQL users, it provides out-of-box experience.
Just like SQL-client, users can enjoy variable sql experiences after
downloading/building a Flink package. In the design, SQL-client is
just a client/part of SQL-gateway.
We can find that, for HiveServer2 and Presto, the similar role is also built-in.

> the Gateway to support multiple Flink versions
I think this idea is good for users, who can use one service to
support multiple Flink versions. But I do not think current design
should support it.
As we know, the API has changed a lot in recent versions of Flink,
and it's so hard to put different versions of Flink's code into one process
 without any class conflicts.
The easiest way to support it would be to use a sub-process model.
For each sub-process, it's the sql gateway we are discussing now.
We can start another project (it can be outside of the Flink project)
to support it.

> For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
>just add it into the class path or remove it from the class path.
The client and the Gateway service may be on different machines.

Best,
Godfrey

LuNing Wang <wa...@gmail.com> 于2022年5月6日周五 17:06写道：
>
> Thanks, Shengkai for driving.  And all for your discussion.
>
>
>
> > intergate the Gateway into the Flink code base
>
> After I talk with Shengkai offline and read the topic `Powering HTAP at
> ByteDance with Apache Flink` of Flink Forward Asia. I think it is better to
> integrate Gateway code into the Flink codebase.
>
>
> In the future, we can add a feature that merges SQL gateway into
> JobManager. We can request JobManager API to directly submit the Flink SQL
> job. It will further improve the performance of Flink OLAP.  In the future,
> the Flink must be a unified engine for batch, stream, and OLAP. The
> Presto/Trino directly requests the master node to submit a job, if so, we
> can reduce Q&M in Flink session mode. Perhaps, the Flink application mode
> can’t merge SQL gateway into JobManager, but Flink OLAP almost always uses
> session mode.
>
> > Gateway to support multiple Flink versions
>
>
> If we will merge the SQL gateway into JobManager, the SQL Gateway itself
> can adapt only one Flink version. We could import a Network Gateway to
> redirect requests to Gateway or JobManager of various versions. Perhaps,
> the network gateway uses other projects, like Apache Kyuubi or Zeppelin,
> etc.
>
> > I don't think that the Gateway is a 'core' function of Flink which should
>
> be included with Flink.
>
> In the production environment, Flink SQL always uses a Gateway. This point
> can be observed in the user email lists and some Flink Forward topics. The
> SQL Gateway is an important infrastructure for big data compute engine. As
> the Flink has not it, many Flink users achieve SQL Gateway in the Apache
> Kyuubi project, but it should be the work of official Flink.
>
> > I think it's fine to move this functionlity to the client rather than
>
> gateway. WDYT?
>
> I agree with the `init-file` option in the client. I think the `init-file`
> functionality in Gateway is NOT important in the first version of Gateway.
> Now, the hive JDBC option ‘initFile’ already has this functionality. After
> SQL Gateway releases and we observe feedback from the community, we maybe
> will discuss this problem again.
>
> Best,
>
> LuNing Wang.
>
>
> Shengkai Fang <fs...@gmail.com> 于2022年5月6日周五 14:34写道：
>
> > Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback
> >
> > > I would like to understand why it's complicated to make the upgrades
> > > problematic
> >
> > I aggree with Jark's point. The API is not very stable in the Flink
> > actually. For example, the Gateway relies on the planner. But in
> > release-1.14 Flink renames the blink planner package. In release-1.15 Flink
> > makes the planner scala free, which means other projects should not
> > directly rely on the planner.
> >
> > >  Does the Flink SQL gateway support submitting a batch job?
> >
> > Of course. In the SQL Gateway, you can just use the sql SET
> > 'execution.runtime-mode' = 'batch' to switch to the batch environment. Then
> > the job you submit later will be executed in the batch mode.
> >
> > > The architecture of the Gateway is in the following graph.
> > Is the TableEnvironment shared for all sessions ?
> >
> > No. Every session has its individual TableEnvironment. I have modified the
> > graph to make everything more clear.
> >
> > > /v1/sessions
> > >> Are both local file and remote file supported for `libs` and `jars`?
> >
> > We don't limit the usage here. But I think we will only support the local
> > file in the next version.
> >
> > >> Does sql gateway support upload files?
> >
> > No. We need a new API to do this. We can collect more user feedback to
> > determine whether we need to implement this feature.
> >
> > >/v1/sessions/:session_handle/configure_session
> > >> Can this api be replaced with `/v1/sessions/:session_handle/statements`
> > ?
> >
> > Actually the API above is different. The
> > `/v1/sessions/:session_handle/configure_session` API uses SQL to configure
> > the environment, which only allows the limited types of SQL. But the
> > `/v1/sessions/:session_handle/statements` has no limitation. I think we'd
> > better use a different API to distinguish these.
> >
> > >/v1/sessions/:session_id/operations/:operation_handle/status
> > >>`:session_id` is a typo, it should be `:session_handdle`
> >
> > Yes. I have fixed the mistake.
> >
> > >/v1/sessions/:session_handle/statements
> > >The statement must be a single command
> >
> > >> Does this api support `begin statement set ... end` or `statement set
> > >> begin ... end`?
> >
> > For BEGIN STATEMENT SET, it will open a buffer in the Session and allows
> > the users to submit the insert statement into the Session later. When the
> > Session receives the END statement, the Gateway will submit the buffered
> > statements.
> >
> > For STATEMENT SET BEGIN ... END, the parser is able to parse the statement.
> > We can treat it as other SQL.
> >
> > >> DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
> >
> > For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
> > just add it into the class path or remove it from the class path. If the
> > jar is the remote jar, we will create a session level directory and
> > download the jar into the directory. When the session closes, it should
> > also clean up all the resources in the session-level directory.
> >
> > I have updated the FLIP to add more info about these.
> >
> > >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > >"type": # string value of LogicalType
> > >> Some LogicalTypes can not be serialized, such as: CharType(0)
> >
> > I think it's about the serialization of the LogicalType. We can follow the
> > behaviour in the LogicalTypeJsonSerializer.
> >
> > > endpoint.protocol
> > >>I think REST is not a kind of protocol[1], but is an architectural style.
> > >> The value should be `HTTP`.
> >
> > I still prefer to use the REST as the value because REST also allows the
> > HTTPS as the protocol. After offline discussion with Godfrey, we think it's
> > better to use the 'endpoint.type' instead.
> >
> > >  Catalog API
> > > ...
> > >> I think we should avoid providing such api, because once catalog api
> > >> is changed or added,
> > >> This class should also be changed. SQL statement is a more general
> > interface.
> >
> > The exposed API is used by the endpoint to organize its required output.
> > The main problem in your plan is that it requires us to parse the data from
> > the RowData, which only contains the basic types. I think it's much more
> > difficult to maintain compared to the current plan that returns structured
> > objects. I think the GatewayService relies on the Catalog but it doesn't
> > mean the GatewayService should expose all the API exposed by the Catalog.
> >
> > > Options
> > >> sql-gateway.session.idle.timeout
> > >> sql-gateway.session.check.interval
> > >> sql-gateway.worker.keepalive.time
> >
> > Okay. I have updated the FLIP about the option names.
> >
> > >why do we need both the rest api and the SQLGatewayService
> > >API, maybe I'm missing something, what's the difference between them?
> >
> > REST API is the user interface. The REST API transforms the request to the
> > invocation of the SQLGatewayService that is the one doing the work. We
> > split the Gateway into the SQLGatewayService and Endpoint(REST API) and its
> > benefit is that all the Endpoints share the same SQLGatewayService.
> >
> > > Is it possible to use one set of rest api to solve all the problems?
> >
> > I think we can't. I don't understand the meaning of all the problems. We
> > can use the REST API to expose all the functionalities in the Gateway side.
> > But many users may have their tools to communicate to the Gateway, which
> > may be based on the HiveServer2 API(thrift api).
> >
> > Best,
> > Shengkai
> >
> >
> >
> >
> >
> >
> >
> > Jingsong Li <ji...@gmail.com> 于2022年5月6日周五 09:16写道：
> >
> > > Thanks Shengkai for driving.  And all for your discussion.
> > >
> > > > The reason why we introduce the gateway with pluggable endpoints is
> > that
> > > many users has their preferences. For example, the HiveServer2 users
> > prefer
> > > to use the gateway with HiveServer2-style API, which has numerous tools.
> > > However, some filnk-native users may prefer to use the REST API.
> > Therefore,
> > > we hope to learn from the Kyuubi's design that expose multiple endpoints
> > > with different API that allow the user to use.
> > >
> > > My understanding is that we need multiple endpoints, But I don't quite
> > > understand why we need both the rest api and the SQLGatewayService
> > > API, maybe I'm missing something, what's the difference between them?
> > > Is it possible to use one set of rest api to solve all the problems?
> > >
> > > > Gateway to support multiple Flink versions
> > >
> > > I think this is a good question to consider.
> > > - First of all, I think it is absolutely impossible for gateway to
> > > support multiple versions of Flink under the current architecture,
> > > because gateway relies on Flink SQL and a lot of SQL compiled and
> > > optimized code is bound to the Flink version.
> > > - The other way is that gateway does not rely on Flink SQL, and each
> > > time a different version of Flink Jar is loaded to compile the job at
> > > once, and frankly speaking, stream jobs actually prefer this model.
> > >
> > > The benefit of gateway support for multiple versions is that it's
> > > really more user-friendly. I've seen cases where users must have
> > > multiple versions existing in a cluster, and if each version needs to
> > > run a gateway, the O&M burden will be heavy.
> > >
> > > > I don't think that the Gateway is a 'core' function of Flink which
> > > should be included with Flink.
> > >
> > > First, I think the Gateway is a 'core' function in Flink.
> > > Why?
> > > I think our architecture should be consistent, which means that Flink
> > > sql-client should use the implementation of gateway, which means that
> > > sql-client depends on gateway.
> > > And sql-client is the basic tool of flink sql, it must exist in flink
> > > repository, otherwise flink sql has no most important entrance.
> > > So, the gateway itself should be our core functionality as well.
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Thu, May 5, 2022 at 10:06 PM Jark Wu <im...@gmail.com> wrote:
> > > >
> > > > Hi Martijn,
> > > >
> > > > Regarding maintaining Gateway inside or outside Flink code base,
> > > > I would like to share my thoughts:
> > > >
> > > > > I would like to understand why it's complicated to make the upgrades
> > > > problematic. Is it because of relying on internal interfaces? If so,
> > > should
> > > > we not consider making them public?
> > > >
> > > > It's not about internal interfaces. Flink itself doesn't provide
> > backward
> > > > compatibility for public APIs.
> > > >
> > > >
> > > > > a) it will not be possible to have separate releases of the Gateway,
> > > > they will be tied to individual Flink releases
> > > > I don't think it's a problem. On the contrary, maintaining a separate
> > > repo
> > > > for Gateway will take a lot of
> > > > extra community efforts, e.g., individual CICD, docs, releases.
> > > >
> > > >
> > > > > b) if you want the Gateway to support multiple Flink versions
> > > > Sorry, I don't see any users requesting this feature for such a long
> > time
> > > > for SQL Gateway.
> > > > Users can build services on Gateway to easily support multi Flink
> > > versions
> > > > (a Gateway for a Flink version).
> > > > It's difficult for Gateway to support multi-version because Flink
> > doesn't
> > > > provide an API that supports backward and forward compatibility.
> > > > If Gateway wants to support multi-version, it has to invent an
> > > > inner-gateway for each version, and Gateway act as a proxy to
> > communicate
> > > > with inner-gateway.
> > > > So you have to have a gateway to couple with the Flink version.
> > > >
> > > > In fact, Gateway is the layer to support multi Flink versions for
> > > > higher-level applications because its API (REST, gRpc) provides
> > backward
> > > > and forward compatibility.
> > > > The gateway itself doesn't need to support multi Flink versions.
> > Besides,
> > > > Trino/Presto also provides servers[1] for each version.
> > > >
> > > >
> > > > > I don't think that the Gateway is a 'core' function of Flink which
> > > should
> > > > be included with Flink.
> > > > Sorry, I can't agree with this. If I remember correctly, Flink SQL has
> > > been
> > > > promoted to first-class citizen for a long time.
> > > > The community also aims to make Flink a truly batch-stream unified
> > > > computing platform, and Gateway would be the entry and center of the
> > > > platform.
> > > > From my point of view, Gateway is a very "core" function and must be
> > > > included in Flink to have better cooperation with SQL and provide an
> > > > out-of-box experience.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > [1]: https://trino.io/download.html
> > > >
> > > > On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com> wrote:
> > > >
> > > > > Hi Shengkai.
> > > > >
> > > > > Thanks for driving the proposal, it's been silent too long.
> > > > >
> > > > > I have a few questions:
> > > > > about the Architecture
> > > > > > The architecture of the Gateway is in the following graph.
> > > > > Is the TableEnvironment shared for all sessions ?
> > > > >
> > > > > about the REST Endpoint
> > > > > > /v1/sessions
> > > > > Are both local file and remote file supported for `libs` and `jars`?
> > > > > Does sql gateway support upload file?
> > > > >
> > > > > >/v1/sessions/:session_handle/configure_session
> > > > > Can this api be replaced with
> > > `/v1/sessions/:session_handle/statements` ?
> > > > >
> > > > > >/v1/sessions/:session_id/operations/:operation_handle/status
> > > > > `:session_id` is a typo, it should be `:session_handdle`
> > > > >
> > > > > >/v1/sessions/:session_handle/statements
> > > > > >The statement must be a single command
> > > > > Does this api support `begin statement set ... end` or `statement set
> > > > > begin ... end`
> > > > >  DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
> > > > >
> > > > >
> > > >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > > > > >"type": # string value of LogicalType
> > > > >  Some LogicalTypes can not be serialized, such as: CharType(0)
> > > > >
> > > > > about Options
> > > > > > endpoint.protocol
> > > > > I think REST is not a kind of protocol[1], but is an architectural
> > > style.
> > > > > The value should be `HTTP`.
> > > > >
> > > > > about SQLGatewayService API
> > > > > >  Catalog API
> > > > > > ...
> > > > > I think we should avoid providing such api, because once catalog api
> > > > > is changed or added,
> > > > > This class should also be changed. SQL statement is a more general
> > > > > interface.
> > > > >
> > > > > > Options
> > > > > > sql-gateway.session.idle.timeout
> > > > > >sql-gateway.session.check.interval
> > > > > >sql-gateway.worker.keepalive.time
> > > > > It's better we can keep the option style as Flink, the level should
> > > > > not be too deep.
> > > > > sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
> > > > > sql-gateway.session.check.interval ->
> > > sql-gateway.session.check-interval
> > > > > sql-gateway.worker.keepalive.time ->
> > sql-gateway.worker.keepalive->time
> > > > >
> > > > > [1] https://restfulapi.net/
> > > > >
> > > > > Best,
> > > > > Godfrey
> > > > >
> > > > > Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
> > > > > >
> > > > > > Hi Shengkai,
> > > > > >
> > > > > > I have another concern about the submission of batch job. Does the
> > > Flink
> > > > > SQL gateway support to submit batch job? In Kyuubi,
> > > BatchProcessBuilder is
> > > > > used to submit batch job. What about the Flink SQL gateway?
> > > > > >
> > > > > > Best regards,
> > > > > > Nicholas Jiang
> > > > > >
> > > > > > On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > > > > > > Hi. Jiang.
> > > > > > >
> > > > > > > Thanks for your feedback！
> > > > > > >
> > > > > > > > Do the public interfaces of GatewayService refer to any
> > service?
> > > > > > >
> > > > > > > We will only expose one GatewayService implementation. We will
> > put
> > > the
> > > > > > > interface into the common package and the developer who wants to
> > > > > implement
> > > > > > > a new endpoint can just rely on the interface package rather than
> > > the
> > > > > > > implementation.
> > > > > > >
> > > > > > > > What's the behavior of SQL Client Gateway working on Yarn or
> > K8S?
> > > > > Does
> > > > > > > the SQL Client Gateway support application or session mode on
> > Yarn?
> > > > > > >
> > > > > > > I think we can support SQL Client Gateway to submit the jobs in
> > > > > > > application/sesison mode.
> > > > > > >
> > > > > > > > Is there any event trigger in the operation state machine?
> > > > > > >
> > > > > > > Yes. I have already updated the content and add more details
> > about
> > > the
> > > > > > > state machine. During the revise, I found that I mix up the two
> > > > > concepts:
> > > > > > > job submission and job execution. In fact, we only control the
> > > > > submission
> > > > > > > mode at the gateway layer. Therefore, we don't need to mapping
> > the
> > > > > > > JobStatus here. If the user expects that the synchronization
> > > behavior
> > > > > is to
> > > > > > > wait for the completion of the job execution before allowing the
> > > next
> > > > > > > statement to be executed, then the Operation lifecycle should
> > also
> > > > > contains
> > > > > > > the job's execution, which means users should set
> > `table.dml-sync`.
> > > > > > >
> > > > > > > > What's the return schema for the public interfaces of
> > > GatewayService?
> > > > > > > Like getTable interface, what's the return value schema?
> > > > > > >
> > > > > > > The API of the GatewayService return the java objects and the
> > > endpoint
> > > > > can
> > > > > > > organize the objects with expected schema. The return results is
> > > also
> > > > > list
> > > > > > > the section ComponetAPI#GatewayService#API. The return type of
> > the
> > > > > > > GatewayService#getTable is `ContextResolvedTable`.
> > > > > > >
> > > > > > > > How does the user get the operation log?
> > > > > > >
> > > > > > > The OperationManager will register the LogAppender before the
> > > Operation
> > > > > > > execution. The Log Appender will hijack the logger and also write
> > > the
> > > > > log
> > > > > > > that related to the Operation to another files. When users wants
> > to
> > > > > fetch
> > > > > > > the Operation log, the GatewayService will read the content in
> > the
> > > > > file and
> > > > > > > return.
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> > > > > > >
> > > > > > > > Hi Shengkai.
> > > > > > > >
> > > > > > > > Thanks for driving the proposal of SQL Client Gateway. I have
> > > some
> > > > > > > > knowledge of Kyuubi and have some questions about the design:
> > > > > > > >
> > > > > > > > 1.Do the public interfaces of GatewayService refer to any
> > > service? If
> > > > > > > > referring to HiveService, does GatewayService need interfaces
> > > like
> > > > > > > > getQueryId etc.
> > > > > > > >
> > > > > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or
> > > K8S?
> > > > > Does
> > > > > > > > the SQL Client Gateway support application or session mode on
> > > Yarn?
> > > > > > > >
> > > > > > > > 3.Is there any event trigger in the operation state machine?
> > > > > > > >
> > > > > > > > 4.What's the return schema for the public interfaces of
> > > > > GatewayService?
> > > > > > > > Like getTable interface, what's the return value schema?
> > > > > > > >
> > > > > > > > 5.How does the user get the operation log?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Nicholas Jiang
> > > > > > > >
> > > > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > > > > > > Hi, Flink developers.
> > > > > > > > >
> > > > > > > > > I want to start a discussion about the FLIP-91: Support Flink
> > > SQL
> > > > > > > > > Gateway[1]. Flink SQL Gateway is a service that allows users
> > to
> > > > > submit
> > > > > > > > and
> > > > > > > > > manage their jobs in the online environment with the
> > pluggable
> > > > > endpoints.
> > > > > > > > > The reason why we introduce the Gateway with pluggable
> > > endpoints
> > > > > is that
> > > > > > > > > many users have their preferences. For example, the
> > HiveServer2
> > > > > users
> > > > > > > > > prefer to use the gateway with HiveServer2-style API, which
> > has
> > > > > numerous
> > > > > > > > > tools. However, some filnk-native users may prefer to use the
> > > REST
> > > > > API.
> > > > > > > > > Therefore, we propose the SQL Gateway with pluggable
> > endpoint.
> > > > > > > > >
> > > > > > > > > In the FLIP, we also propose the REST endpoint, which has the
> > > > > similar
> > > > > > > > > APIs compared to the gateway in the
> > > > > ververica/flink-sql-gateway[2]. At
> > > > > > > > the
> > > > > > > > > last, we discuss how to use the SQL Client to submit the
> > > statement
> > > > > to the
> > > > > > > > > Gateway with the REST API.
> > > > > > > > >
> > > > > > > > > I am glad that you can give some feedback about FLIP-91.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shengkai
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > > > > > > [2] https://github.com/ververica/flink-sql-gateway
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[LuNing Wang <wa...@gmail.com>]

Thanks, Shengkai for driving.  And all for your discussion.



> intergate the Gateway into the Flink code base

After I talk with Shengkai offline and read the topic `Powering HTAP at
ByteDance with Apache Flink` of Flink Forward Asia. I think it is better to
integrate Gateway code into the Flink codebase.


In the future, we can add a feature that merges SQL gateway into
JobManager. We can request JobManager API to directly submit the Flink SQL
job. It will further improve the performance of Flink OLAP.  In the future,
the Flink must be a unified engine for batch, stream, and OLAP. The
Presto/Trino directly requests the master node to submit a job, if so, we
can reduce Q&M in Flink session mode. Perhaps, the Flink application mode
can’t merge SQL gateway into JobManager, but Flink OLAP almost always uses
session mode.

> Gateway to support multiple Flink versions


If we will merge the SQL gateway into JobManager, the SQL Gateway itself
can adapt only one Flink version. We could import a Network Gateway to
redirect requests to Gateway or JobManager of various versions. Perhaps,
the network gateway uses other projects, like Apache Kyuubi or Zeppelin,
etc.

> I don't think that the Gateway is a 'core' function of Flink which should

be included with Flink.

In the production environment, Flink SQL always uses a Gateway. This point
can be observed in the user email lists and some Flink Forward topics. The
SQL Gateway is an important infrastructure for big data compute engine. As
the Flink has not it, many Flink users achieve SQL Gateway in the Apache
Kyuubi project, but it should be the work of official Flink.

> I think it's fine to move this functionlity to the client rather than

gateway. WDYT?

I agree with the `init-file` option in the client. I think the `init-file`
functionality in Gateway is NOT important in the first version of Gateway.
Now, the hive JDBC option ‘initFile’ already has this functionality. After
SQL Gateway releases and we observe feedback from the community, we maybe
will discuss this problem again.

Best,

LuNing Wang.


Shengkai Fang <fs...@gmail.com> 于2022年5月6日周五 14:34写道：

> Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback
>
> > I would like to understand why it's complicated to make the upgrades
> > problematic
>
> I aggree with Jark's point. The API is not very stable in the Flink
> actually. For example, the Gateway relies on the planner. But in
> release-1.14 Flink renames the blink planner package. In release-1.15 Flink
> makes the planner scala free, which means other projects should not
> directly rely on the planner.
>
> >  Does the Flink SQL gateway support submitting a batch job?
>
> Of course. In the SQL Gateway, you can just use the sql SET
> 'execution.runtime-mode' = 'batch' to switch to the batch environment. Then
> the job you submit later will be executed in the batch mode.
>
> > The architecture of the Gateway is in the following graph.
> Is the TableEnvironment shared for all sessions ?
>
> No. Every session has its individual TableEnvironment. I have modified the
> graph to make everything more clear.
>
> > /v1/sessions
> >> Are both local file and remote file supported for `libs` and `jars`?
>
> We don't limit the usage here. But I think we will only support the local
> file in the next version.
>
> >> Does sql gateway support upload files?
>
> No. We need a new API to do this. We can collect more user feedback to
> determine whether we need to implement this feature.
>
> >/v1/sessions/:session_handle/configure_session
> >> Can this api be replaced with `/v1/sessions/:session_handle/statements`
> ?
>
> Actually the API above is different. The
> `/v1/sessions/:session_handle/configure_session` API uses SQL to configure
> the environment, which only allows the limited types of SQL. But the
> `/v1/sessions/:session_handle/statements` has no limitation. I think we'd
> better use a different API to distinguish these.
>
> >/v1/sessions/:session_id/operations/:operation_handle/status
> >>`:session_id` is a typo, it should be `:session_handdle`
>
> Yes. I have fixed the mistake.
>
> >/v1/sessions/:session_handle/statements
> >The statement must be a single command
>
> >> Does this api support `begin statement set ... end` or `statement set
> >> begin ... end`?
>
> For BEGIN STATEMENT SET, it will open a buffer in the Session and allows
> the users to submit the insert statement into the Session later. When the
> Session receives the END statement, the Gateway will submit the buffered
> statements.
>
> For STATEMENT SET BEGIN ... END, the parser is able to parse the statement.
> We can treat it as other SQL.
>
> >> DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
>
> For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
> just add it into the class path or remove it from the class path. If the
> jar is the remote jar, we will create a session level directory and
> download the jar into the directory. When the session closes, it should
> also clean up all the resources in the session-level directory.
>
> I have updated the FLIP to add more info about these.
>
> >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> >"type": # string value of LogicalType
> >> Some LogicalTypes can not be serialized, such as: CharType(0)
>
> I think it's about the serialization of the LogicalType. We can follow the
> behaviour in the LogicalTypeJsonSerializer.
>
> > endpoint.protocol
> >>I think REST is not a kind of protocol[1], but is an architectural style.
> >> The value should be `HTTP`.
>
> I still prefer to use the REST as the value because REST also allows the
> HTTPS as the protocol. After offline discussion with Godfrey, we think it's
> better to use the 'endpoint.type' instead.
>
> >  Catalog API
> > ...
> >> I think we should avoid providing such api, because once catalog api
> >> is changed or added,
> >> This class should also be changed. SQL statement is a more general
> interface.
>
> The exposed API is used by the endpoint to organize its required output.
> The main problem in your plan is that it requires us to parse the data from
> the RowData, which only contains the basic types. I think it's much more
> difficult to maintain compared to the current plan that returns structured
> objects. I think the GatewayService relies on the Catalog but it doesn't
> mean the GatewayService should expose all the API exposed by the Catalog.
>
> > Options
> >> sql-gateway.session.idle.timeout
> >> sql-gateway.session.check.interval
> >> sql-gateway.worker.keepalive.time
>
> Okay. I have updated the FLIP about the option names.
>
> >why do we need both the rest api and the SQLGatewayService
> >API, maybe I'm missing something, what's the difference between them?
>
> REST API is the user interface. The REST API transforms the request to the
> invocation of the SQLGatewayService that is the one doing the work. We
> split the Gateway into the SQLGatewayService and Endpoint(REST API) and its
> benefit is that all the Endpoints share the same SQLGatewayService.
>
> > Is it possible to use one set of rest api to solve all the problems?
>
> I think we can't. I don't understand the meaning of all the problems. We
> can use the REST API to expose all the functionalities in the Gateway side.
> But many users may have their tools to communicate to the Gateway, which
> may be based on the HiveServer2 API(thrift api).
>
> Best,
> Shengkai
>
>
>
>
>
>
>
> Jingsong Li <ji...@gmail.com> 于2022年5月6日周五 09:16写道：
>
> > Thanks Shengkai for driving.  And all for your discussion.
> >
> > > The reason why we introduce the gateway with pluggable endpoints is
> that
> > many users has their preferences. For example, the HiveServer2 users
> prefer
> > to use the gateway with HiveServer2-style API, which has numerous tools.
> > However, some filnk-native users may prefer to use the REST API.
> Therefore,
> > we hope to learn from the Kyuubi's design that expose multiple endpoints
> > with different API that allow the user to use.
> >
> > My understanding is that we need multiple endpoints, But I don't quite
> > understand why we need both the rest api and the SQLGatewayService
> > API, maybe I'm missing something, what's the difference between them?
> > Is it possible to use one set of rest api to solve all the problems?
> >
> > > Gateway to support multiple Flink versions
> >
> > I think this is a good question to consider.
> > - First of all, I think it is absolutely impossible for gateway to
> > support multiple versions of Flink under the current architecture,
> > because gateway relies on Flink SQL and a lot of SQL compiled and
> > optimized code is bound to the Flink version.
> > - The other way is that gateway does not rely on Flink SQL, and each
> > time a different version of Flink Jar is loaded to compile the job at
> > once, and frankly speaking, stream jobs actually prefer this model.
> >
> > The benefit of gateway support for multiple versions is that it's
> > really more user-friendly. I've seen cases where users must have
> > multiple versions existing in a cluster, and if each version needs to
> > run a gateway, the O&M burden will be heavy.
> >
> > > I don't think that the Gateway is a 'core' function of Flink which
> > should be included with Flink.
> >
> > First, I think the Gateway is a 'core' function in Flink.
> > Why?
> > I think our architecture should be consistent, which means that Flink
> > sql-client should use the implementation of gateway, which means that
> > sql-client depends on gateway.
> > And sql-client is the basic tool of flink sql, it must exist in flink
> > repository, otherwise flink sql has no most important entrance.
> > So, the gateway itself should be our core functionality as well.
> >
> > Best,
> > Jingsong
> >
> > On Thu, May 5, 2022 at 10:06 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > > Hi Martijn,
> > >
> > > Regarding maintaining Gateway inside or outside Flink code base,
> > > I would like to share my thoughts:
> > >
> > > > I would like to understand why it's complicated to make the upgrades
> > > problematic. Is it because of relying on internal interfaces? If so,
> > should
> > > we not consider making them public?
> > >
> > > It's not about internal interfaces. Flink itself doesn't provide
> backward
> > > compatibility for public APIs.
> > >
> > >
> > > > a) it will not be possible to have separate releases of the Gateway,
> > > they will be tied to individual Flink releases
> > > I don't think it's a problem. On the contrary, maintaining a separate
> > repo
> > > for Gateway will take a lot of
> > > extra community efforts, e.g., individual CICD, docs, releases.
> > >
> > >
> > > > b) if you want the Gateway to support multiple Flink versions
> > > Sorry, I don't see any users requesting this feature for such a long
> time
> > > for SQL Gateway.
> > > Users can build services on Gateway to easily support multi Flink
> > versions
> > > (a Gateway for a Flink version).
> > > It's difficult for Gateway to support multi-version because Flink
> doesn't
> > > provide an API that supports backward and forward compatibility.
> > > If Gateway wants to support multi-version, it has to invent an
> > > inner-gateway for each version, and Gateway act as a proxy to
> communicate
> > > with inner-gateway.
> > > So you have to have a gateway to couple with the Flink version.
> > >
> > > In fact, Gateway is the layer to support multi Flink versions for
> > > higher-level applications because its API (REST, gRpc) provides
> backward
> > > and forward compatibility.
> > > The gateway itself doesn't need to support multi Flink versions.
> Besides,
> > > Trino/Presto also provides servers[1] for each version.
> > >
> > >
> > > > I don't think that the Gateway is a 'core' function of Flink which
> > should
> > > be included with Flink.
> > > Sorry, I can't agree with this. If I remember correctly, Flink SQL has
> > been
> > > promoted to first-class citizen for a long time.
> > > The community also aims to make Flink a truly batch-stream unified
> > > computing platform, and Gateway would be the entry and center of the
> > > platform.
> > > From my point of view, Gateway is a very "core" function and must be
> > > included in Flink to have better cooperation with SQL and provide an
> > > out-of-box experience.
> > >
> > > Best,
> > > Jark
> > >
> > > [1]: https://trino.io/download.html
> > >
> > > On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com> wrote:
> > >
> > > > Hi Shengkai.
> > > >
> > > > Thanks for driving the proposal, it's been silent too long.
> > > >
> > > > I have a few questions:
> > > > about the Architecture
> > > > > The architecture of the Gateway is in the following graph.
> > > > Is the TableEnvironment shared for all sessions ?
> > > >
> > > > about the REST Endpoint
> > > > > /v1/sessions
> > > > Are both local file and remote file supported for `libs` and `jars`?
> > > > Does sql gateway support upload file?
> > > >
> > > > >/v1/sessions/:session_handle/configure_session
> > > > Can this api be replaced with
> > `/v1/sessions/:session_handle/statements` ?
> > > >
> > > > >/v1/sessions/:session_id/operations/:operation_handle/status
> > > > `:session_id` is a typo, it should be `:session_handdle`
> > > >
> > > > >/v1/sessions/:session_handle/statements
> > > > >The statement must be a single command
> > > > Does this api support `begin statement set ... end` or `statement set
> > > > begin ... end`
> > > >  DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
> > > >
> > > >
> > >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > > > >"type": # string value of LogicalType
> > > >  Some LogicalTypes can not be serialized, such as: CharType(0)
> > > >
> > > > about Options
> > > > > endpoint.protocol
> > > > I think REST is not a kind of protocol[1], but is an architectural
> > style.
> > > > The value should be `HTTP`.
> > > >
> > > > about SQLGatewayService API
> > > > >  Catalog API
> > > > > ...
> > > > I think we should avoid providing such api, because once catalog api
> > > > is changed or added,
> > > > This class should also be changed. SQL statement is a more general
> > > > interface.
> > > >
> > > > > Options
> > > > > sql-gateway.session.idle.timeout
> > > > >sql-gateway.session.check.interval
> > > > >sql-gateway.worker.keepalive.time
> > > > It's better we can keep the option style as Flink, the level should
> > > > not be too deep.
> > > > sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
> > > > sql-gateway.session.check.interval ->
> > sql-gateway.session.check-interval
> > > > sql-gateway.worker.keepalive.time ->
> sql-gateway.worker.keepalive->time
> > > >
> > > > [1] https://restfulapi.net/
> > > >
> > > > Best,
> > > > Godfrey
> > > >
> > > > Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
> > > > >
> > > > > Hi Shengkai,
> > > > >
> > > > > I have another concern about the submission of batch job. Does the
> > Flink
> > > > SQL gateway support to submit batch job? In Kyuubi,
> > BatchProcessBuilder is
> > > > used to submit batch job. What about the Flink SQL gateway?
> > > > >
> > > > > Best regards,
> > > > > Nicholas Jiang
> > > > >
> > > > > On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > > > > > Hi. Jiang.
> > > > > >
> > > > > > Thanks for your feedback！
> > > > > >
> > > > > > > Do the public interfaces of GatewayService refer to any
> service?
> > > > > >
> > > > > > We will only expose one GatewayService implementation. We will
> put
> > the
> > > > > > interface into the common package and the developer who wants to
> > > > implement
> > > > > > a new endpoint can just rely on the interface package rather than
> > the
> > > > > > implementation.
> > > > > >
> > > > > > > What's the behavior of SQL Client Gateway working on Yarn or
> K8S?
> > > > Does
> > > > > > the SQL Client Gateway support application or session mode on
> Yarn?
> > > > > >
> > > > > > I think we can support SQL Client Gateway to submit the jobs in
> > > > > > application/sesison mode.
> > > > > >
> > > > > > > Is there any event trigger in the operation state machine?
> > > > > >
> > > > > > Yes. I have already updated the content and add more details
> about
> > the
> > > > > > state machine. During the revise, I found that I mix up the two
> > > > concepts:
> > > > > > job submission and job execution. In fact, we only control the
> > > > submission
> > > > > > mode at the gateway layer. Therefore, we don't need to mapping
> the
> > > > > > JobStatus here. If the user expects that the synchronization
> > behavior
> > > > is to
> > > > > > wait for the completion of the job execution before allowing the
> > next
> > > > > > statement to be executed, then the Operation lifecycle should
> also
> > > > contains
> > > > > > the job's execution, which means users should set
> `table.dml-sync`.
> > > > > >
> > > > > > > What's the return schema for the public interfaces of
> > GatewayService?
> > > > > > Like getTable interface, what's the return value schema?
> > > > > >
> > > > > > The API of the GatewayService return the java objects and the
> > endpoint
> > > > can
> > > > > > organize the objects with expected schema. The return results is
> > also
> > > > list
> > > > > > the section ComponetAPI#GatewayService#API. The return type of
> the
> > > > > > GatewayService#getTable is `ContextResolvedTable`.
> > > > > >
> > > > > > > How does the user get the operation log?
> > > > > >
> > > > > > The OperationManager will register the LogAppender before the
> > Operation
> > > > > > execution. The Log Appender will hijack the logger and also write
> > the
> > > > log
> > > > > > that related to the Operation to another files. When users wants
> to
> > > > fetch
> > > > > > the Operation log, the GatewayService will read the content in
> the
> > > > file and
> > > > > > return.
> > > > > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> > > > > >
> > > > > > > Hi Shengkai.
> > > > > > >
> > > > > > > Thanks for driving the proposal of SQL Client Gateway. I have
> > some
> > > > > > > knowledge of Kyuubi and have some questions about the design:
> > > > > > >
> > > > > > > 1.Do the public interfaces of GatewayService refer to any
> > service? If
> > > > > > > referring to HiveService, does GatewayService need interfaces
> > like
> > > > > > > getQueryId etc.
> > > > > > >
> > > > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or
> > K8S?
> > > > Does
> > > > > > > the SQL Client Gateway support application or session mode on
> > Yarn?
> > > > > > >
> > > > > > > 3.Is there any event trigger in the operation state machine?
> > > > > > >
> > > > > > > 4.What's the return schema for the public interfaces of
> > > > GatewayService?
> > > > > > > Like getTable interface, what's the return value schema?
> > > > > > >
> > > > > > > 5.How does the user get the operation log?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Nicholas Jiang
> > > > > > >
> > > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > > > > > Hi, Flink developers.
> > > > > > > >
> > > > > > > > I want to start a discussion about the FLIP-91: Support Flink
> > SQL
> > > > > > > > Gateway[1]. Flink SQL Gateway is a service that allows users
> to
> > > > submit
> > > > > > > and
> > > > > > > > manage their jobs in the online environment with the
> pluggable
> > > > endpoints.
> > > > > > > > The reason why we introduce the Gateway with pluggable
> > endpoints
> > > > is that
> > > > > > > > many users have their preferences. For example, the
> HiveServer2
> > > > users
> > > > > > > > prefer to use the gateway with HiveServer2-style API, which
> has
> > > > numerous
> > > > > > > > tools. However, some filnk-native users may prefer to use the
> > REST
> > > > API.
> > > > > > > > Therefore, we propose the SQL Gateway with pluggable
> endpoint.
> > > > > > > >
> > > > > > > > In the FLIP, we also propose the REST endpoint, which has the
> > > > similar
> > > > > > > > APIs compared to the gateway in the
> > > > ververica/flink-sql-gateway[2]. At
> > > > > > > the
> > > > > > > > last, we discuss how to use the SQL Client to submit the
> > statement
> > > > to the
> > > > > > > > Gateway with the REST API.
> > > > > > > >
> > > > > > > > I am glad that you can give some feedback about FLIP-91.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shengkai
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > > > > > [2] https://github.com/ververica/flink-sql-gateway
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Thanks Martijn, Nicholas, Godfrey, Jark and Jingsong feedback

> I would like to understand why it's complicated to make the upgrades
> problematic

I aggree with Jark's point. The API is not very stable in the Flink
actually. For example, the Gateway relies on the planner. But in
release-1.14 Flink renames the blink planner package. In release-1.15 Flink
makes the planner scala free, which means other projects should not
directly rely on the planner.

>  Does the Flink SQL gateway support submitting a batch job?

Of course. In the SQL Gateway, you can just use the sql SET
'execution.runtime-mode' = 'batch' to switch to the batch environment. Then
the job you submit later will be executed in the batch mode.

> The architecture of the Gateway is in the following graph.
Is the TableEnvironment shared for all sessions ?

No. Every session has its individual TableEnvironment. I have modified the
graph to make everything more clear.

> /v1/sessions
>> Are both local file and remote file supported for `libs` and `jars`?

We don't limit the usage here. But I think we will only support the local
file in the next version.

>> Does sql gateway support upload files?

No. We need a new API to do this. We can collect more user feedback to
determine whether we need to implement this feature.

>/v1/sessions/:session_handle/configure_session
>> Can this api be replaced with `/v1/sessions/:session_handle/statements` ?

Actually the API above is different. The
`/v1/sessions/:session_handle/configure_session` API uses SQL to configure
the environment, which only allows the limited types of SQL. But the
`/v1/sessions/:session_handle/statements` has no limitation. I think we'd
better use a different API to distinguish these.

>/v1/sessions/:session_id/operations/:operation_handle/status
>>`:session_id` is a typo, it should be `:session_handdle`

Yes. I have fixed the mistake.

>/v1/sessions/:session_handle/statements
>The statement must be a single command

>> Does this api support `begin statement set ... end` or `statement set
>> begin ... end`?

For BEGIN STATEMENT SET, it will open a buffer in the Session and allows
the users to submit the insert statement into the Session later. When the
Session receives the END statement, the Gateway will submit the buffered
statements.

For STATEMENT SET BEGIN ... END, the parser is able to parse the statement.
We can treat it as other SQL.

>> DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?

For ADD JAR/REMOVE JAR, if the jar is in the local environment, we will
just add it into the class path or remove it from the class path. If the
jar is the remote jar, we will create a session level directory and
download the jar into the directory. When the session closes, it should
also clean up all the resources in the session-level directory.

I have updated the FLIP to add more info about these.

>/v1/sessions/:session_handle/operations/:operation_handle/result/:token
>"type": # string value of LogicalType
>> Some LogicalTypes can not be serialized, such as: CharType(0)

I think it's about the serialization of the LogicalType. We can follow the
behaviour in the LogicalTypeJsonSerializer.

> endpoint.protocol
>>I think REST is not a kind of protocol[1], but is an architectural style.
>> The value should be `HTTP`.

I still prefer to use the REST as the value because REST also allows the
HTTPS as the protocol. After offline discussion with Godfrey, we think it's
better to use the 'endpoint.type' instead.

>  Catalog API
> ...
>> I think we should avoid providing such api, because once catalog api
>> is changed or added,
>> This class should also be changed. SQL statement is a more general
interface.

The exposed API is used by the endpoint to organize its required output.
The main problem in your plan is that it requires us to parse the data from
the RowData, which only contains the basic types. I think it's much more
difficult to maintain compared to the current plan that returns structured
objects. I think the GatewayService relies on the Catalog but it doesn't
mean the GatewayService should expose all the API exposed by the Catalog.

> Options
>> sql-gateway.session.idle.timeout
>> sql-gateway.session.check.interval
>> sql-gateway.worker.keepalive.time

Okay. I have updated the FLIP about the option names.

>why do we need both the rest api and the SQLGatewayService
>API, maybe I'm missing something, what's the difference between them?

REST API is the user interface. The REST API transforms the request to the
invocation of the SQLGatewayService that is the one doing the work. We
split the Gateway into the SQLGatewayService and Endpoint(REST API) and its
benefit is that all the Endpoints share the same SQLGatewayService.

> Is it possible to use one set of rest api to solve all the problems?

I think we can't. I don't understand the meaning of all the problems. We
can use the REST API to expose all the functionalities in the Gateway side.
But many users may have their tools to communicate to the Gateway, which
may be based on the HiveServer2 API(thrift api).

Best,
Shengkai







Jingsong Li <ji...@gmail.com> 于2022年5月6日周五 09:16写道：

> Thanks Shengkai for driving.  And all for your discussion.
>
> > The reason why we introduce the gateway with pluggable endpoints is that
> many users has their preferences. For example, the HiveServer2 users prefer
> to use the gateway with HiveServer2-style API, which has numerous tools.
> However, some filnk-native users may prefer to use the REST API. Therefore,
> we hope to learn from the Kyuubi's design that expose multiple endpoints
> with different API that allow the user to use.
>
> My understanding is that we need multiple endpoints, But I don't quite
> understand why we need both the rest api and the SQLGatewayService
> API, maybe I'm missing something, what's the difference between them?
> Is it possible to use one set of rest api to solve all the problems?
>
> > Gateway to support multiple Flink versions
>
> I think this is a good question to consider.
> - First of all, I think it is absolutely impossible for gateway to
> support multiple versions of Flink under the current architecture,
> because gateway relies on Flink SQL and a lot of SQL compiled and
> optimized code is bound to the Flink version.
> - The other way is that gateway does not rely on Flink SQL, and each
> time a different version of Flink Jar is loaded to compile the job at
> once, and frankly speaking, stream jobs actually prefer this model.
>
> The benefit of gateway support for multiple versions is that it's
> really more user-friendly. I've seen cases where users must have
> multiple versions existing in a cluster, and if each version needs to
> run a gateway, the O&M burden will be heavy.
>
> > I don't think that the Gateway is a 'core' function of Flink which
> should be included with Flink.
>
> First, I think the Gateway is a 'core' function in Flink.
> Why?
> I think our architecture should be consistent, which means that Flink
> sql-client should use the implementation of gateway, which means that
> sql-client depends on gateway.
> And sql-client is the basic tool of flink sql, it must exist in flink
> repository, otherwise flink sql has no most important entrance.
> So, the gateway itself should be our core functionality as well.
>
> Best,
> Jingsong
>
> On Thu, May 5, 2022 at 10:06 PM Jark Wu <im...@gmail.com> wrote:
> >
> > Hi Martijn,
> >
> > Regarding maintaining Gateway inside or outside Flink code base,
> > I would like to share my thoughts:
> >
> > > I would like to understand why it's complicated to make the upgrades
> > problematic. Is it because of relying on internal interfaces? If so,
> should
> > we not consider making them public?
> >
> > It's not about internal interfaces. Flink itself doesn't provide backward
> > compatibility for public APIs.
> >
> >
> > > a) it will not be possible to have separate releases of the Gateway,
> > they will be tied to individual Flink releases
> > I don't think it's a problem. On the contrary, maintaining a separate
> repo
> > for Gateway will take a lot of
> > extra community efforts, e.g., individual CICD, docs, releases.
> >
> >
> > > b) if you want the Gateway to support multiple Flink versions
> > Sorry, I don't see any users requesting this feature for such a long time
> > for SQL Gateway.
> > Users can build services on Gateway to easily support multi Flink
> versions
> > (a Gateway for a Flink version).
> > It's difficult for Gateway to support multi-version because Flink doesn't
> > provide an API that supports backward and forward compatibility.
> > If Gateway wants to support multi-version, it has to invent an
> > inner-gateway for each version, and Gateway act as a proxy to communicate
> > with inner-gateway.
> > So you have to have a gateway to couple with the Flink version.
> >
> > In fact, Gateway is the layer to support multi Flink versions for
> > higher-level applications because its API (REST, gRpc) provides backward
> > and forward compatibility.
> > The gateway itself doesn't need to support multi Flink versions. Besides,
> > Trino/Presto also provides servers[1] for each version.
> >
> >
> > > I don't think that the Gateway is a 'core' function of Flink which
> should
> > be included with Flink.
> > Sorry, I can't agree with this. If I remember correctly, Flink SQL has
> been
> > promoted to first-class citizen for a long time.
> > The community also aims to make Flink a truly batch-stream unified
> > computing platform, and Gateway would be the entry and center of the
> > platform.
> > From my point of view, Gateway is a very "core" function and must be
> > included in Flink to have better cooperation with SQL and provide an
> > out-of-box experience.
> >
> > Best,
> > Jark
> >
> > [1]: https://trino.io/download.html
> >
> > On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com> wrote:
> >
> > > Hi Shengkai.
> > >
> > > Thanks for driving the proposal, it's been silent too long.
> > >
> > > I have a few questions:
> > > about the Architecture
> > > > The architecture of the Gateway is in the following graph.
> > > Is the TableEnvironment shared for all sessions ?
> > >
> > > about the REST Endpoint
> > > > /v1/sessions
> > > Are both local file and remote file supported for `libs` and `jars`?
> > > Does sql gateway support upload file?
> > >
> > > >/v1/sessions/:session_handle/configure_session
> > > Can this api be replaced with
> `/v1/sessions/:session_handle/statements` ?
> > >
> > > >/v1/sessions/:session_id/operations/:operation_handle/status
> > > `:session_id` is a typo, it should be `:session_handdle`
> > >
> > > >/v1/sessions/:session_handle/statements
> > > >The statement must be a single command
> > > Does this api support `begin statement set ... end` or `statement set
> > > begin ... end`
> > >  DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
> > >
> > >
> >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > > >"type": # string value of LogicalType
> > >  Some LogicalTypes can not be serialized, such as: CharType(0)
> > >
> > > about Options
> > > > endpoint.protocol
> > > I think REST is not a kind of protocol[1], but is an architectural
> style.
> > > The value should be `HTTP`.
> > >
> > > about SQLGatewayService API
> > > >  Catalog API
> > > > ...
> > > I think we should avoid providing such api, because once catalog api
> > > is changed or added,
> > > This class should also be changed. SQL statement is a more general
> > > interface.
> > >
> > > > Options
> > > > sql-gateway.session.idle.timeout
> > > >sql-gateway.session.check.interval
> > > >sql-gateway.worker.keepalive.time
> > > It's better we can keep the option style as Flink, the level should
> > > not be too deep.
> > > sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
> > > sql-gateway.session.check.interval ->
> sql-gateway.session.check-interval
> > > sql-gateway.worker.keepalive.time -> sql-gateway.worker.keepalive->time
> > >
> > > [1] https://restfulapi.net/
> > >
> > > Best,
> > > Godfrey
> > >
> > > Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
> > > >
> > > > Hi Shengkai,
> > > >
> > > > I have another concern about the submission of batch job. Does the
> Flink
> > > SQL gateway support to submit batch job? In Kyuubi,
> BatchProcessBuilder is
> > > used to submit batch job. What about the Flink SQL gateway?
> > > >
> > > > Best regards,
> > > > Nicholas Jiang
> > > >
> > > > On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > > > > Hi. Jiang.
> > > > >
> > > > > Thanks for your feedback！
> > > > >
> > > > > > Do the public interfaces of GatewayService refer to any service?
> > > > >
> > > > > We will only expose one GatewayService implementation. We will put
> the
> > > > > interface into the common package and the developer who wants to
> > > implement
> > > > > a new endpoint can just rely on the interface package rather than
> the
> > > > > implementation.
> > > > >
> > > > > > What's the behavior of SQL Client Gateway working on Yarn or K8S?
> > > Does
> > > > > the SQL Client Gateway support application or session mode on Yarn?
> > > > >
> > > > > I think we can support SQL Client Gateway to submit the jobs in
> > > > > application/sesison mode.
> > > > >
> > > > > > Is there any event trigger in the operation state machine?
> > > > >
> > > > > Yes. I have already updated the content and add more details about
> the
> > > > > state machine. During the revise, I found that I mix up the two
> > > concepts:
> > > > > job submission and job execution. In fact, we only control the
> > > submission
> > > > > mode at the gateway layer. Therefore, we don't need to mapping the
> > > > > JobStatus here. If the user expects that the synchronization
> behavior
> > > is to
> > > > > wait for the completion of the job execution before allowing the
> next
> > > > > statement to be executed, then the Operation lifecycle should also
> > > contains
> > > > > the job's execution, which means users should set `table.dml-sync`.
> > > > >
> > > > > > What's the return schema for the public interfaces of
> GatewayService?
> > > > > Like getTable interface, what's the return value schema?
> > > > >
> > > > > The API of the GatewayService return the java objects and the
> endpoint
> > > can
> > > > > organize the objects with expected schema. The return results is
> also
> > > list
> > > > > the section ComponetAPI#GatewayService#API. The return type of the
> > > > > GatewayService#getTable is `ContextResolvedTable`.
> > > > >
> > > > > > How does the user get the operation log?
> > > > >
> > > > > The OperationManager will register the LogAppender before the
> Operation
> > > > > execution. The Log Appender will hijack the logger and also write
> the
> > > log
> > > > > that related to the Operation to another files. When users wants to
> > > fetch
> > > > > the Operation log, the GatewayService will read the content in the
> > > file and
> > > > > return.
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> > > > >
> > > > > > Hi Shengkai.
> > > > > >
> > > > > > Thanks for driving the proposal of SQL Client Gateway. I have
> some
> > > > > > knowledge of Kyuubi and have some questions about the design:
> > > > > >
> > > > > > 1.Do the public interfaces of GatewayService refer to any
> service? If
> > > > > > referring to HiveService, does GatewayService need interfaces
> like
> > > > > > getQueryId etc.
> > > > > >
> > > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or
> K8S?
> > > Does
> > > > > > the SQL Client Gateway support application or session mode on
> Yarn?
> > > > > >
> > > > > > 3.Is there any event trigger in the operation state machine?
> > > > > >
> > > > > > 4.What's the return schema for the public interfaces of
> > > GatewayService?
> > > > > > Like getTable interface, what's the return value schema?
> > > > > >
> > > > > > 5.How does the user get the operation log?
> > > > > >
> > > > > > Thanks,
> > > > > > Nicholas Jiang
> > > > > >
> > > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > > > > Hi, Flink developers.
> > > > > > >
> > > > > > > I want to start a discussion about the FLIP-91: Support Flink
> SQL
> > > > > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
> > > submit
> > > > > > and
> > > > > > > manage their jobs in the online environment with the pluggable
> > > endpoints.
> > > > > > > The reason why we introduce the Gateway with pluggable
> endpoints
> > > is that
> > > > > > > many users have their preferences. For example, the HiveServer2
> > > users
> > > > > > > prefer to use the gateway with HiveServer2-style API, which has
> > > numerous
> > > > > > > tools. However, some filnk-native users may prefer to use the
> REST
> > > API.
> > > > > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > > > > > >
> > > > > > > In the FLIP, we also propose the REST endpoint, which has the
> > > similar
> > > > > > > APIs compared to the gateway in the
> > > ververica/flink-sql-gateway[2]. At
> > > > > > the
> > > > > > > last, we discuss how to use the SQL Client to submit the
> statement
> > > to the
> > > > > > > Gateway with the REST API.
> > > > > > >
> > > > > > > I am glad that you can give some feedback about FLIP-91.
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > > > > [2] https://github.com/ververica/flink-sql-gateway
> > > > > > >
> > > > > >
> > > > >
> > >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jingsong Li <ji...@gmail.com>]

Thanks Shengkai for driving.  And all for your discussion.

> The reason why we introduce the gateway with pluggable endpoints is that many users has their preferences. For example, the HiveServer2 users prefer to use the gateway with HiveServer2-style API, which has numerous tools. However, some filnk-native users may prefer to use the REST API. Therefore, we hope to learn from the Kyuubi's design that expose multiple endpoints with different API that allow the user to use.

My understanding is that we need multiple endpoints, But I don't quite
understand why we need both the rest api and the SQLGatewayService
API, maybe I'm missing something, what's the difference between them?
Is it possible to use one set of rest api to solve all the problems?

> Gateway to support multiple Flink versions

I think this is a good question to consider.
- First of all, I think it is absolutely impossible for gateway to
support multiple versions of Flink under the current architecture,
because gateway relies on Flink SQL and a lot of SQL compiled and
optimized code is bound to the Flink version.
- The other way is that gateway does not rely on Flink SQL, and each
time a different version of Flink Jar is loaded to compile the job at
once, and frankly speaking, stream jobs actually prefer this model.

The benefit of gateway support for multiple versions is that it's
really more user-friendly. I've seen cases where users must have
multiple versions existing in a cluster, and if each version needs to
run a gateway, the O&M burden will be heavy.

> I don't think that the Gateway is a 'core' function of Flink which should be included with Flink.

First, I think the Gateway is a 'core' function in Flink.
Why?
I think our architecture should be consistent, which means that Flink
sql-client should use the implementation of gateway, which means that
sql-client depends on gateway.
And sql-client is the basic tool of flink sql, it must exist in flink
repository, otherwise flink sql has no most important entrance.
So, the gateway itself should be our core functionality as well.

Best,
Jingsong

On Thu, May 5, 2022 at 10:06 PM Jark Wu <im...@gmail.com> wrote:
>
> Hi Martijn,
>
> Regarding maintaining Gateway inside or outside Flink code base,
> I would like to share my thoughts:
>
> > I would like to understand why it's complicated to make the upgrades
> problematic. Is it because of relying on internal interfaces? If so, should
> we not consider making them public?
>
> It's not about internal interfaces. Flink itself doesn't provide backward
> compatibility for public APIs.
>
>
> > a) it will not be possible to have separate releases of the Gateway,
> they will be tied to individual Flink releases
> I don't think it's a problem. On the contrary, maintaining a separate repo
> for Gateway will take a lot of
> extra community efforts, e.g., individual CICD, docs, releases.
>
>
> > b) if you want the Gateway to support multiple Flink versions
> Sorry, I don't see any users requesting this feature for such a long time
> for SQL Gateway.
> Users can build services on Gateway to easily support multi Flink versions
> (a Gateway for a Flink version).
> It's difficult for Gateway to support multi-version because Flink doesn't
> provide an API that supports backward and forward compatibility.
> If Gateway wants to support multi-version, it has to invent an
> inner-gateway for each version, and Gateway act as a proxy to communicate
> with inner-gateway.
> So you have to have a gateway to couple with the Flink version.
>
> In fact, Gateway is the layer to support multi Flink versions for
> higher-level applications because its API (REST, gRpc) provides backward
> and forward compatibility.
> The gateway itself doesn't need to support multi Flink versions. Besides,
> Trino/Presto also provides servers[1] for each version.
>
>
> > I don't think that the Gateway is a 'core' function of Flink which should
> be included with Flink.
> Sorry, I can't agree with this. If I remember correctly, Flink SQL has been
> promoted to first-class citizen for a long time.
> The community also aims to make Flink a truly batch-stream unified
> computing platform, and Gateway would be the entry and center of the
> platform.
> From my point of view, Gateway is a very "core" function and must be
> included in Flink to have better cooperation with SQL and provide an
> out-of-box experience.
>
> Best,
> Jark
>
> [1]: https://trino.io/download.html
>
> On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com> wrote:
>
> > Hi Shengkai.
> >
> > Thanks for driving the proposal, it's been silent too long.
> >
> > I have a few questions:
> > about the Architecture
> > > The architecture of the Gateway is in the following graph.
> > Is the TableEnvironment shared for all sessions ?
> >
> > about the REST Endpoint
> > > /v1/sessions
> > Are both local file and remote file supported for `libs` and `jars`?
> > Does sql gateway support upload file?
> >
> > >/v1/sessions/:session_handle/configure_session
> > Can this api be replaced with `/v1/sessions/:session_handle/statements` ?
> >
> > >/v1/sessions/:session_id/operations/:operation_handle/status
> > `:session_id` is a typo, it should be `:session_handdle`
> >
> > >/v1/sessions/:session_handle/statements
> > >The statement must be a single command
> > Does this api support `begin statement set ... end` or `statement set
> > begin ... end`
> >  DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
> >
> > >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> > >"type": # string value of LogicalType
> >  Some LogicalTypes can not be serialized, such as: CharType(0)
> >
> > about Options
> > > endpoint.protocol
> > I think REST is not a kind of protocol[1], but is an architectural style.
> > The value should be `HTTP`.
> >
> > about SQLGatewayService API
> > >  Catalog API
> > > ...
> > I think we should avoid providing such api, because once catalog api
> > is changed or added,
> > This class should also be changed. SQL statement is a more general
> > interface.
> >
> > > Options
> > > sql-gateway.session.idle.timeout
> > >sql-gateway.session.check.interval
> > >sql-gateway.worker.keepalive.time
> > It's better we can keep the option style as Flink, the level should
> > not be too deep.
> > sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
> > sql-gateway.session.check.interval -> sql-gateway.session.check-interval
> > sql-gateway.worker.keepalive.time -> sql-gateway.worker.keepalive->time
> >
> > [1] https://restfulapi.net/
> >
> > Best,
> > Godfrey
> >
> > Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
> > >
> > > Hi Shengkai,
> > >
> > > I have another concern about the submission of batch job. Does the Flink
> > SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is
> > used to submit batch job. What about the Flink SQL gateway?
> > >
> > > Best regards,
> > > Nicholas Jiang
> > >
> > > On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > > > Hi. Jiang.
> > > >
> > > > Thanks for your feedback！
> > > >
> > > > > Do the public interfaces of GatewayService refer to any service?
> > > >
> > > > We will only expose one GatewayService implementation. We will put the
> > > > interface into the common package and the developer who wants to
> > implement
> > > > a new endpoint can just rely on the interface package rather than the
> > > > implementation.
> > > >
> > > > > What's the behavior of SQL Client Gateway working on Yarn or K8S?
> > Does
> > > > the SQL Client Gateway support application or session mode on Yarn?
> > > >
> > > > I think we can support SQL Client Gateway to submit the jobs in
> > > > application/sesison mode.
> > > >
> > > > > Is there any event trigger in the operation state machine?
> > > >
> > > > Yes. I have already updated the content and add more details about the
> > > > state machine. During the revise, I found that I mix up the two
> > concepts:
> > > > job submission and job execution. In fact, we only control the
> > submission
> > > > mode at the gateway layer. Therefore, we don't need to mapping the
> > > > JobStatus here. If the user expects that the synchronization behavior
> > is to
> > > > wait for the completion of the job execution before allowing the next
> > > > statement to be executed, then the Operation lifecycle should also
> > contains
> > > > the job's execution, which means users should set `table.dml-sync`.
> > > >
> > > > > What's the return schema for the public interfaces of GatewayService?
> > > > Like getTable interface, what's the return value schema?
> > > >
> > > > The API of the GatewayService return the java objects and the endpoint
> > can
> > > > organize the objects with expected schema. The return results is also
> > list
> > > > the section ComponetAPI#GatewayService#API. The return type of the
> > > > GatewayService#getTable is `ContextResolvedTable`.
> > > >
> > > > > How does the user get the operation log?
> > > >
> > > > The OperationManager will register the LogAppender before the Operation
> > > > execution. The Log Appender will hijack the logger and also write the
> > log
> > > > that related to the Operation to another files. When users wants to
> > fetch
> > > > the Operation log, the GatewayService will read the content in the
> > file and
> > > > return.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > >
> > > >
> > > >
> > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> > > >
> > > > > Hi Shengkai.
> > > > >
> > > > > Thanks for driving the proposal of SQL Client Gateway. I have some
> > > > > knowledge of Kyuubi and have some questions about the design:
> > > > >
> > > > > 1.Do the public interfaces of GatewayService refer to any service? If
> > > > > referring to HiveService, does GatewayService need interfaces like
> > > > > getQueryId etc.
> > > > >
> > > > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S?
> > Does
> > > > > the SQL Client Gateway support application or session mode on Yarn?
> > > > >
> > > > > 3.Is there any event trigger in the operation state machine?
> > > > >
> > > > > 4.What's the return schema for the public interfaces of
> > GatewayService?
> > > > > Like getTable interface, what's the return value schema?
> > > > >
> > > > > 5.How does the user get the operation log?
> > > > >
> > > > > Thanks,
> > > > > Nicholas Jiang
> > > > >
> > > > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > > > Hi, Flink developers.
> > > > > >
> > > > > > I want to start a discussion about the FLIP-91: Support Flink SQL
> > > > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
> > submit
> > > > > and
> > > > > > manage their jobs in the online environment with the pluggable
> > endpoints.
> > > > > > The reason why we introduce the Gateway with pluggable endpoints
> > is that
> > > > > > many users have their preferences. For example, the HiveServer2
> > users
> > > > > > prefer to use the gateway with HiveServer2-style API, which has
> > numerous
> > > > > > tools. However, some filnk-native users may prefer to use the REST
> > API.
> > > > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > > > > >
> > > > > > In the FLIP, we also propose the REST endpoint, which has the
> > similar
> > > > > > APIs compared to the gateway in the
> > ververica/flink-sql-gateway[2]. At
> > > > > the
> > > > > > last, we discuss how to use the SQL Client to submit the statement
> > to the
> > > > > > Gateway with the REST API.
> > > > > >
> > > > > > I am glad that you can give some feedback about FLIP-91.
> > > > > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > > > [2] https://github.com/ververica/flink-sql-gateway
> > > > > >
> > > > >
> > > >
> >

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Jark Wu <im...@gmail.com>]

Hi Martijn,

Regarding maintaining Gateway inside or outside Flink code base,
I would like to share my thoughts:

> I would like to understand why it's complicated to make the upgrades
problematic. Is it because of relying on internal interfaces? If so, should
we not consider making them public?

It's not about internal interfaces. Flink itself doesn't provide backward
compatibility for public APIs.


> a) it will not be possible to have separate releases of the Gateway,
they will be tied to individual Flink releases
I don't think it's a problem. On the contrary, maintaining a separate repo
for Gateway will take a lot of
extra community efforts, e.g., individual CICD, docs, releases.


> b) if you want the Gateway to support multiple Flink versions
Sorry, I don't see any users requesting this feature for such a long time
for SQL Gateway.
Users can build services on Gateway to easily support multi Flink versions
(a Gateway for a Flink version).
It's difficult for Gateway to support multi-version because Flink doesn't
provide an API that supports backward and forward compatibility.
If Gateway wants to support multi-version, it has to invent an
inner-gateway for each version, and Gateway act as a proxy to communicate
with inner-gateway.
So you have to have a gateway to couple with the Flink version.

In fact, Gateway is the layer to support multi Flink versions for
higher-level applications because its API (REST, gRpc) provides backward
and forward compatibility.
The gateway itself doesn't need to support multi Flink versions. Besides,
Trino/Presto also provides servers[1] for each version.


> I don't think that the Gateway is a 'core' function of Flink which should
be included with Flink.
Sorry, I can't agree with this. If I remember correctly, Flink SQL has been
promoted to first-class citizen for a long time.
The community also aims to make Flink a truly batch-stream unified
computing platform, and Gateway would be the entry and center of the
platform.
From my point of view, Gateway is a very "core" function and must be
included in Flink to have better cooperation with SQL and provide an
out-of-box experience.

Best,
Jark

[1]: https://trino.io/download.html

On Thu, 5 May 2022 at 19:57, godfrey he <go...@gmail.com> wrote:

> Hi Shengkai.
>
> Thanks for driving the proposal, it's been silent too long.
>
> I have a few questions:
> about the Architecture
> > The architecture of the Gateway is in the following graph.
> Is the TableEnvironment shared for all sessions ?
>
> about the REST Endpoint
> > /v1/sessions
> Are both local file and remote file supported for `libs` and `jars`?
> Does sql gateway support upload file?
>
> >/v1/sessions/:session_handle/configure_session
> Can this api be replaced with `/v1/sessions/:session_handle/statements` ?
>
> >/v1/sessions/:session_id/operations/:operation_handle/status
> `:session_id` is a typo, it should be `:session_handdle`
>
> >/v1/sessions/:session_handle/statements
> >The statement must be a single command
> Does this api support `begin statement set ... end` or `statement set
> begin ... end`
>  DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?
>
> >/v1/sessions/:session_handle/operations/:operation_handle/result/:token
> >"type": # string value of LogicalType
>  Some LogicalTypes can not be serialized, such as: CharType(0)
>
> about Options
> > endpoint.protocol
> I think REST is not a kind of protocol[1], but is an architectural style.
> The value should be `HTTP`.
>
> about SQLGatewayService API
> >  Catalog API
> > ...
> I think we should avoid providing such api, because once catalog api
> is changed or added,
> This class should also be changed. SQL statement is a more general
> interface.
>
> > Options
> > sql-gateway.session.idle.timeout
> >sql-gateway.session.check.interval
> >sql-gateway.worker.keepalive.time
> It's better we can keep the option style as Flink, the level should
> not be too deep.
> sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
> sql-gateway.session.check.interval -> sql-gateway.session.check-interval
> sql-gateway.worker.keepalive.time -> sql-gateway.worker.keepalive->time
>
> [1] https://restfulapi.net/
>
> Best,
> Godfrey
>
> Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
> >
> > Hi Shengkai,
> >
> > I have another concern about the submission of batch job. Does the Flink
> SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is
> used to submit batch job. What about the Flink SQL gateway?
> >
> > Best regards,
> > Nicholas Jiang
> >
> > On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > > Hi. Jiang.
> > >
> > > Thanks for your feedback！
> > >
> > > > Do the public interfaces of GatewayService refer to any service?
> > >
> > > We will only expose one GatewayService implementation. We will put the
> > > interface into the common package and the developer who wants to
> implement
> > > a new endpoint can just rely on the interface package rather than the
> > > implementation.
> > >
> > > > What's the behavior of SQL Client Gateway working on Yarn or K8S?
> Does
> > > the SQL Client Gateway support application or session mode on Yarn?
> > >
> > > I think we can support SQL Client Gateway to submit the jobs in
> > > application/sesison mode.
> > >
> > > > Is there any event trigger in the operation state machine?
> > >
> > > Yes. I have already updated the content and add more details about the
> > > state machine. During the revise, I found that I mix up the two
> concepts:
> > > job submission and job execution. In fact, we only control the
> submission
> > > mode at the gateway layer. Therefore, we don't need to mapping the
> > > JobStatus here. If the user expects that the synchronization behavior
> is to
> > > wait for the completion of the job execution before allowing the next
> > > statement to be executed, then the Operation lifecycle should also
> contains
> > > the job's execution, which means users should set `table.dml-sync`.
> > >
> > > > What's the return schema for the public interfaces of GatewayService?
> > > Like getTable interface, what's the return value schema?
> > >
> > > The API of the GatewayService return the java objects and the endpoint
> can
> > > organize the objects with expected schema. The return results is also
> list
> > > the section ComponetAPI#GatewayService#API. The return type of the
> > > GatewayService#getTable is `ContextResolvedTable`.
> > >
> > > > How does the user get the operation log?
> > >
> > > The OperationManager will register the LogAppender before the Operation
> > > execution. The Log Appender will hijack the logger and also write the
> log
> > > that related to the Operation to another files. When users wants to
> fetch
> > > the Operation log, the GatewayService will read the content in the
> file and
> > > return.
> > >
> > > Best,
> > > Shengkai
> > >
> > >
> > >
> > >
> > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> > >
> > > > Hi Shengkai.
> > > >
> > > > Thanks for driving the proposal of SQL Client Gateway. I have some
> > > > knowledge of Kyuubi and have some questions about the design:
> > > >
> > > > 1.Do the public interfaces of GatewayService refer to any service? If
> > > > referring to HiveService, does GatewayService need interfaces like
> > > > getQueryId etc.
> > > >
> > > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S?
> Does
> > > > the SQL Client Gateway support application or session mode on Yarn?
> > > >
> > > > 3.Is there any event trigger in the operation state machine?
> > > >
> > > > 4.What's the return schema for the public interfaces of
> GatewayService?
> > > > Like getTable interface, what's the return value schema?
> > > >
> > > > 5.How does the user get the operation log?
> > > >
> > > > Thanks,
> > > > Nicholas Jiang
> > > >
> > > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > > Hi, Flink developers.
> > > > >
> > > > > I want to start a discussion about the FLIP-91: Support Flink SQL
> > > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
> submit
> > > > and
> > > > > manage their jobs in the online environment with the pluggable
> endpoints.
> > > > > The reason why we introduce the Gateway with pluggable endpoints
> is that
> > > > > many users have their preferences. For example, the HiveServer2
> users
> > > > > prefer to use the gateway with HiveServer2-style API, which has
> numerous
> > > > > tools. However, some filnk-native users may prefer to use the REST
> API.
> > > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > > > >
> > > > > In the FLIP, we also propose the REST endpoint, which has the
> similar
> > > > > APIs compared to the gateway in the
> ververica/flink-sql-gateway[2]. At
> > > > the
> > > > > last, we discuss how to use the SQL Client to submit the statement
> to the
> > > > > Gateway with the REST API.
> > > > >
> > > > > I am glad that you can give some feedback about FLIP-91.
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > [1]
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > > [2] https://github.com/ververica/flink-sql-gateway
> > > > >
> > > >
> > >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[godfrey he <go...@gmail.com>]

Hi Shengkai.

Thanks for driving the proposal, it's been silent too long.

I have a few questions:
about the Architecture
> The architecture of the Gateway is in the following graph.
Is the TableEnvironment shared for all sessions ?

about the REST Endpoint
> /v1/sessions
Are both local file and remote file supported for `libs` and `jars`?
Does sql gateway support upload file?

>/v1/sessions/:session_handle/configure_session
Can this api be replaced with `/v1/sessions/:session_handle/statements` ?

>/v1/sessions/:session_id/operations/:operation_handle/status
`:session_id` is a typo, it should be `:session_handdle`

>/v1/sessions/:session_handle/statements
>The statement must be a single command
Does this api support `begin statement set ... end` or `statement set
begin ... end`
 DO `ADD JAR`, `REMOVE JAR` support ? If yes, how to manage the jars?

>/v1/sessions/:session_handle/operations/:operation_handle/result/:token
>"type": # string value of LogicalType
 Some LogicalTypes can not be serialized, such as: CharType(0)

about Options
> endpoint.protocol
I think REST is not a kind of protocol[1], but is an architectural style.
The value should be `HTTP`.

about SQLGatewayService API
>  Catalog API
> ...
I think we should avoid providing such api, because once catalog api
is changed or added,
This class should also be changed. SQL statement is a more general interface.

> Options
> sql-gateway.session.idle.timeout
>sql-gateway.session.check.interval
>sql-gateway.worker.keepalive.time
It's better we can keep the option style as Flink, the level should
not be too deep.
sql-gateway.session.idle.timeout -> sql-gateway.session.idle-timeout
sql-gateway.session.check.interval -> sql-gateway.session.check-interval
sql-gateway.worker.keepalive.time -> sql-gateway.worker.keepalive->time

[1] https://restfulapi.net/

Best,
Godfrey

Nicholas Jiang <ni...@apache.org> 于2022年5月5日周四 14:58写道：
>
> Hi Shengkai,
>
> I have another concern about the submission of batch job. Does the Flink SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is used to submit batch job. What about the Flink SQL gateway?
>
> Best regards,
> Nicholas Jiang
>
> On 2022/04/24 03:28:36 Shengkai Fang wrote:
> > Hi. Jiang.
> >
> > Thanks for your feedback！
> >
> > > Do the public interfaces of GatewayService refer to any service?
> >
> > We will only expose one GatewayService implementation. We will put the
> > interface into the common package and the developer who wants to implement
> > a new endpoint can just rely on the interface package rather than the
> > implementation.
> >
> > > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> > the SQL Client Gateway support application or session mode on Yarn?
> >
> > I think we can support SQL Client Gateway to submit the jobs in
> > application/sesison mode.
> >
> > > Is there any event trigger in the operation state machine?
> >
> > Yes. I have already updated the content and add more details about the
> > state machine. During the revise, I found that I mix up the two concepts:
> > job submission and job execution. In fact, we only control the submission
> > mode at the gateway layer. Therefore, we don't need to mapping the
> > JobStatus here. If the user expects that the synchronization behavior is to
> > wait for the completion of the job execution before allowing the next
> > statement to be executed, then the Operation lifecycle should also contains
> > the job's execution, which means users should set `table.dml-sync`.
> >
> > > What's the return schema for the public interfaces of GatewayService?
> > Like getTable interface, what's the return value schema?
> >
> > The API of the GatewayService return the java objects and the endpoint can
> > organize the objects with expected schema. The return results is also list
> > the section ComponetAPI#GatewayService#API. The return type of the
> > GatewayService#getTable is `ContextResolvedTable`.
> >
> > > How does the user get the operation log?
> >
> > The OperationManager will register the LogAppender before the Operation
> > execution. The Log Appender will hijack the logger and also write the log
> > that related to the Operation to another files. When users wants to fetch
> > the Operation log, the GatewayService will read the content in the file and
> > return.
> >
> > Best,
> > Shengkai
> >
> >
> >
> >
> > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> >
> > > Hi Shengkai.
> > >
> > > Thanks for driving the proposal of SQL Client Gateway. I have some
> > > knowledge of Kyuubi and have some questions about the design:
> > >
> > > 1.Do the public interfaces of GatewayService refer to any service? If
> > > referring to HiveService, does GatewayService need interfaces like
> > > getQueryId etc.
> > >
> > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> > > the SQL Client Gateway support application or session mode on Yarn?
> > >
> > > 3.Is there any event trigger in the operation state machine?
> > >
> > > 4.What's the return schema for the public interfaces of GatewayService?
> > > Like getTable interface, what's the return value schema?
> > >
> > > 5.How does the user get the operation log?
> > >
> > > Thanks,
> > > Nicholas Jiang
> > >
> > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > Hi, Flink developers.
> > > >
> > > > I want to start a discussion about the FLIP-91: Support Flink SQL
> > > > Gateway[1]. Flink SQL Gateway is a service that allows users to submit
> > > and
> > > > manage their jobs in the online environment with the pluggable endpoints.
> > > > The reason why we introduce the Gateway with pluggable endpoints is that
> > > > many users have their preferences. For example, the HiveServer2 users
> > > > prefer to use the gateway with HiveServer2-style API, which has numerous
> > > > tools. However, some filnk-native users may prefer to use the REST API.
> > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > > >
> > > > In the FLIP, we also propose the REST endpoint, which has the similar
> > > > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At
> > > the
> > > > last, we discuss how to use the SQL Client to submit the statement to the
> > > > Gateway with the REST API.
> > > >
> > > > I am glad that you can give some feedback about FLIP-91.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > [1]
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > [2] https://github.com/ververica/flink-sql-gateway
> > > >
> > >
> >

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Nicholas Jiang <ni...@apache.org>]

Hi Shengkai,

I have another concern about the submission of batch job. Does the Flink SQL gateway support to submit batch job? In Kyuubi, BatchProcessBuilder is used to submit batch job. What about the Flink SQL gateway?

Best regards,
Nicholas Jiang

On 2022/04/24 03:28:36 Shengkai Fang wrote:
> Hi. Jiang.
> 
> Thanks for your feedback！
> 
> > Do the public interfaces of GatewayService refer to any service?
> 
> We will only expose one GatewayService implementation. We will put the
> interface into the common package and the developer who wants to implement
> a new endpoint can just rely on the interface package rather than the
> implementation.
> 
> > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> the SQL Client Gateway support application or session mode on Yarn?
> 
> I think we can support SQL Client Gateway to submit the jobs in
> application/sesison mode.
> 
> > Is there any event trigger in the operation state machine?
> 
> Yes. I have already updated the content and add more details about the
> state machine. During the revise, I found that I mix up the two concepts:
> job submission and job execution. In fact, we only control the submission
> mode at the gateway layer. Therefore, we don't need to mapping the
> JobStatus here. If the user expects that the synchronization behavior is to
> wait for the completion of the job execution before allowing the next
> statement to be executed, then the Operation lifecycle should also contains
> the job's execution, which means users should set `table.dml-sync`.
> 
> > What's the return schema for the public interfaces of GatewayService?
> Like getTable interface, what's the return value schema?
> 
> The API of the GatewayService return the java objects and the endpoint can
> organize the objects with expected schema. The return results is also list
> the section ComponetAPI#GatewayService#API. The return type of the
> GatewayService#getTable is `ContextResolvedTable`.
> 
> > How does the user get the operation log?
> 
> The OperationManager will register the LogAppender before the Operation
> execution. The Log Appender will hijack the logger and also write the log
> that related to the Operation to another files. When users wants to fetch
> the Operation log, the GatewayService will read the content in the file and
> return.
> 
> Best,
> Shengkai
> 
> 
> 
> 
> Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> 
> > Hi Shengkai.
> >
> > Thanks for driving the proposal of SQL Client Gateway. I have some
> > knowledge of Kyuubi and have some questions about the design:
> >
> > 1.Do the public interfaces of GatewayService refer to any service? If
> > referring to HiveService, does GatewayService need interfaces like
> > getQueryId etc.
> >
> > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> > the SQL Client Gateway support application or session mode on Yarn?
> >
> > 3.Is there any event trigger in the operation state machine?
> >
> > 4.What's the return schema for the public interfaces of GatewayService?
> > Like getTable interface, what's the return value schema?
> >
> > 5.How does the user get the operation log?
> >
> > Thanks,
> > Nicholas Jiang
> >
> > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > Hi, Flink developers.
> > >
> > > I want to start a discussion about the FLIP-91: Support Flink SQL
> > > Gateway[1]. Flink SQL Gateway is a service that allows users to submit
> > and
> > > manage their jobs in the online environment with the pluggable endpoints.
> > > The reason why we introduce the Gateway with pluggable endpoints is that
> > > many users have their preferences. For example, the HiveServer2 users
> > > prefer to use the gateway with HiveServer2-style API, which has numerous
> > > tools. However, some filnk-native users may prefer to use the REST API.
> > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > >
> > > In the FLIP, we also propose the REST endpoint, which has the similar
> > > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At
> > the
> > > last, we discuss how to use the SQL Client to submit the statement to the
> > > Gateway with the REST API.
> > >
> > > I am glad that you can give some feedback about FLIP-91.
> > >
> > > Best,
> > > Shengkai
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > [2] https://github.com/ververica/flink-sql-gateway
> > >
> >
> 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Martijn Visser <ma...@apache.org>]

Hi Shengkai,

> Agreed. The FLIP mainly focus on the Gateway. I think it's better to
rename the name to the "Support SQL Gateway". WDYT?

+1

> I think it's better to intergate the Gateway into the Flink code base.
The reason behind is
>  1. The Gateway relies on the Flink implementation,  I think we'd better
to maintain it inside the Flink. It really takes us much time to upgrade
the sql-gateway in ververica repo to the latest Flink version.
> 2. The Gateway is important to the Flink itself. Many users needs the
Gateway to manage the Flink SQL jobs. Actually Hive, Spark both have its
Gateway in its code base.

I would like to understand why it's complicated to make the upgrades
problematic. Is it because of relying on internal interfaces? If so, should
we not consider making them public?

A downside I see with integrating the Gateway into the Flink codebase is
that a) it will not be possible to have separate releases of the Gateway,
they will be tied to individual Flink releases and b) if you want the
Gateway to support multiple Flink versions, I can see that becoming
complicated in Flink's release branching and support mechanism. For
example, what if you have a Gateway released with Flink in Flink 1.16 and
Flink 1.17, which both support Flink 1.10 up to their latest version. Then
you encounter a bug in the implementation for Flink 1.12: that means that
you have to create multiple fixes, in multiple branches, and then release
multiple new Flink versions. I don't think that the Gateway is a 'core'
function of Flink which should be included with Flink. There are a lot of
users who use the DataStream, Table or Python as their implementation
layer. They all don't need this extra capability (even though you could
argue that in the future it would be nice to have something similar for
Python).

> Because the Gateway itself relies on the Flink inner implementation...I
think we can just use one Gateway per versions. Users can manage the
gateway with other utils.

I've left my comment above because I was going through each argument one by
one, but this was my assumption already: we should not rely on internal
interfaces for capability we want to support as a community. We should then
make these interfaces public.

> After I read FLIP-91[1], I want to add an init-file option. Its
functionality is the same as option '-i' of Flink SQL Client.

I don't think that an init-file option should be added to the SQL Gateway.
+1 for keeping that in the client, not the Gateway.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser


On Fri, 29 Apr 2022 at 04:26, Shengkai Fang <fs...@gmail.com> wrote:

> Hi Marijn and LuNing.
>
> Thanks for your feedback!
>
> > The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
> which would be used by Flink's SQL Client (or other applications)?
>
> Agreed. The FLIP mainly focus on the Gateway. I think it's better to rename
> the name to the "Support SQL Gateway". WDYT?
>
> > From a user perspective, I would have expected that we start with the
> REST endpoint before explaining how we would integrate this into Flink. Now
> it's quite hard to first understand what we want to offer to users and if
> that will be sufficient for a first version.
>
> emmm. Considering that api is basically the operation of some concepts, is
> it better to introduce the core concepts first? But I agree you are right
> that we should start with the RESt endpoint. I reorganize the content to
> introduce the REST first in the public interfaces.
>
> > With Flink 1.15, we're introducing an OpenAPI specification. Can we
> also do this straight away for the REST Gateway?
>
> Yes. We will organize the related APIs into OpenAPI specification.
>
> >Should we introduce the REST Gateway as part of Flink's main repository?
> >Wouldn't we be better off to maintain this in a separate repository under
> >ASF?
>
> I think it's better to intergate the Gateway into the Flink code base. The
> reason behind is
>
> 1. The Gateway relies on the Flink implementation,  I think we'd better to
> maintain it inside the Flink. It really takes us much time to upgrade the
> sql-gateway in ververica repo to the latest Flink version.
>
> 2. The Gateway is important to the Flink itself. Many users needs the
> Gateway to manage the Flink SQL jobs. Actually Hive, Spark both have its
> Gateway in its code base.
>
> But I think it's fine to put other utils, e.g. JDBC under the ASF.
>
> > Ideally you would like to be able to support multiple Flink versions
> > with one version of the REST Gateway I think?
>
> > Users can upgrade a large number of Flink jobs versions gradually in a
> Gateway service.
>
> Because the Gateway itself relies on the Flink inner implementation...I
> think we can just use one Gateway per versions. Users can manage the
> gateway with other utils.
>
> >There's no mention of Batch or Streaming in this concept. If I recall
> >correctly, the current Flink SQL Gateway can only support Batch. How will
> >we support Streaming?
>
> > I can imagine that if a user wants to use a REST Gateway, there's also a
> > strong need to combine this with a Catalog.
>
> Yes. I add a section about the Usage of the Gateway. Users can use the SQL
> do everything in the Gateway, including
> - configure the execution parameter, including exectuion mode
> - manage the metadata with DDL, e.g. register catalog
> - submit the job
> ...
>
> >Will there be any requirement with JDBC, as there currently is?
>
> In the FLIP-223, we implement the HiveServer2 endpint. Users can use the
> hive jdbc to connect to the Flink SQL Gateway.
>
> > Shall we name this option `sql-gateway.session.init-file` and write it
> into
> the FLIP-91?
>
> Actually we already supports the -i parameters in the sql client. What's
> more, Hive also supports the -i parameter in the client side[1].
> I think it's fine to move this functionlity to the client rather than
> gateway. WDYT?
>
> [1]
>
> https://github.com/apache/hive/blob/c3fa88a1b7d1475f44383fca913aecf9c664bab0/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L321
>
> Best,
> Shengkai
>
>
>
>
>
> LuNing Wang <wa...@gmail.com> 于2022年4月28日周四 10:04写道：
>
> > > * Should we introduce the REST Gateway as part of Flink's main
> > repository?
> > Wouldn't we be better off to maintain this in a separate repository under
> > ASF? Ideally you would like to be able to support multiple Flink versions
> > with one version of the REST Gateway I think?
> >
> > We would be better off maintaining this in a separate repository. It is
> > important to support multiple Flink versions. Users can upgrade a large
> > number of Flink jobs versions gradually in a Gateway service.
> >
> > LuNing Wang <wa...@gmail.com> 于2022年4月27日周三 17:54写道：
> >
> > > Hi ShengKai,
> > >
> > > After I read FLIP-91[1], I want to add an init-file option. Its
> > > functionality is the same as option '-i' of Flink SQL Client.
> > >
> > > When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by
> > > this option after SQL Gateway starts every time.
> > >
> > > Shall we name this option `sql-gateway.session.init-file` and write it
> > > into the FLIP-91?
> > >
> > > Best regards,
> > >
> > > LuNing Wang
> > >
> > > [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91
> > >
> > > Martijn Visser <ma...@ververica.com> 于2022年4月26日周二 20:32写道：
> > >
> > >> Hi Shengkai,
> > >>
> > >> Thanks for opening this discussion. I did a first brief pass over the
> > FLIP
> > >> and I have a couple of questions/remarks:
> > >>
> > >> * The FLIP is called "SQL Client Gateway", but isn't this a REST
> Gateway
> > >> which would be used by Flink's SQL Client (or other applications)?
> > >>
> > >> * From a user perspective, I would have expected that we start with
> the
> > >> REST endpoint before explaining how we would integrate this into
> Flink.
> > >> Now
> > >> it's quite hard to first understand what we want to offer to users and
> > if
> > >> that will be sufficient for a first version.
> > >>
> > >> * With Flink 1.15, we're introducing an OpenAPI specification [1]. Can
> > we
> > >> also do this straight away for the REST Gateway?
> > >>
> > >> * Should we introduce the REST Gateway as part of Flink's main
> > repository?
> > >> Wouldn't we be better off to maintain this in a separate repository
> > under
> > >> ASF? Ideally you would like to be able to support multiple Flink
> > versions
> > >> with one version of the REST Gateway I think?
> > >>
> > >> * There's no mention of Batch or Streaming in this concept. If I
> recall
> > >> correctly, the current Flink SQL Gateway can only support Batch. How
> > will
> > >> we support Streaming? Will there be any requirement with JDBC, as
> there
> > >> currently is?
> > >>
> > >> * I can imagine that if a user wants to use a REST Gateway, there's
> > also a
> > >> strong need to combine this with a Catalog. Do you think this should
> be
> > >> part of this FLIP?
> > >>
> > >> Best regards,
> > >>
> > >> Martijn Visser
> > >> https://twitter.com/MartijnVisser82
> > >> https://github.com/MartijnVisser
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager
> > >>
> > >> On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fs...@gmail.com>
> wrote:
> > >>
> > >> > Hi. Jiang.
> > >> >
> > >> > Thanks for your feedback！
> > >> >
> > >> > > Do the public interfaces of GatewayService refer to any service?
> > >> >
> > >> > We will only expose one GatewayService implementation. We will put
> the
> > >> > interface into the common package and the developer who wants to
> > >> implement
> > >> > a new endpoint can just rely on the interface package rather than
> the
> > >> > implementation.
> > >> >
> > >> > > What's the behavior of SQL Client Gateway working on Yarn or K8S?
> > Does
> > >> > the SQL Client Gateway support application or session mode on Yarn?
> > >> >
> > >> > I think we can support SQL Client Gateway to submit the jobs in
> > >> > application/sesison mode.
> > >> >
> > >> > > Is there any event trigger in the operation state machine?
> > >> >
> > >> > Yes. I have already updated the content and add more details about
> the
> > >> > state machine. During the revise, I found that I mix up the two
> > >> concepts:
> > >> > job submission and job execution. In fact, we only control the
> > >> submission
> > >> > mode at the gateway layer. Therefore, we don't need to mapping the
> > >> > JobStatus here. If the user expects that the synchronization
> behavior
> > >> is to
> > >> > wait for the completion of the job execution before allowing the
> next
> > >> > statement to be executed, then the Operation lifecycle should also
> > >> contains
> > >> > the job's execution, which means users should set `table.dml-sync`.
> > >> >
> > >> > > What's the return schema for the public interfaces of
> > GatewayService?
> > >> > Like getTable interface, what's the return value schema?
> > >> >
> > >> > The API of the GatewayService return the java objects and the
> endpoint
> > >> can
> > >> > organize the objects with expected schema. The return results is
> also
> > >> list
> > >> > the section ComponetAPI#GatewayService#API. The return type of the
> > >> > GatewayService#getTable is `ContextResolvedTable`.
> > >> >
> > >> > > How does the user get the operation log?
> > >> >
> > >> > The OperationManager will register the LogAppender before the
> > Operation
> > >> > execution. The Log Appender will hijack the logger and also write
> the
> > >> log
> > >> > that related to the Operation to another files. When users wants to
> > >> fetch
> > >> > the Operation log, the GatewayService will read the content in the
> > file
> > >> and
> > >> > return.
> > >> >
> > >> > Best,
> > >> > Shengkai
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> > >> >
> > >> > > Hi Shengkai.
> > >> > >
> > >> > > Thanks for driving the proposal of SQL Client Gateway. I have some
> > >> > > knowledge of Kyuubi and have some questions about the design:
> > >> > >
> > >> > > 1.Do the public interfaces of GatewayService refer to any service?
> > If
> > >> > > referring to HiveService, does GatewayService need interfaces like
> > >> > > getQueryId etc.
> > >> > >
> > >> > > 2.What's the behavior of SQL Client Gateway working on Yarn or
> K8S?
> > >> Does
> > >> > > the SQL Client Gateway support application or session mode on
> Yarn?
> > >> > >
> > >> > > 3.Is there any event trigger in the operation state machine?
> > >> > >
> > >> > > 4.What's the return schema for the public interfaces of
> > >> GatewayService?
> > >> > > Like getTable interface, what's the return value schema?
> > >> > >
> > >> > > 5.How does the user get the operation log?
> > >> > >
> > >> > > Thanks,
> > >> > > Nicholas Jiang
> > >> > >
> > >> > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > >> > > > Hi, Flink developers.
> > >> > > >
> > >> > > > I want to start a discussion about the FLIP-91: Support Flink
> SQL
> > >> > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
> > >> submit
> > >> > > and
> > >> > > > manage their jobs in the online environment with the pluggable
> > >> > endpoints.
> > >> > > > The reason why we introduce the Gateway with pluggable endpoints
> > is
> > >> > that
> > >> > > > many users have their preferences. For example, the HiveServer2
> > >> users
> > >> > > > prefer to use the gateway with HiveServer2-style API, which has
> > >> > numerous
> > >> > > > tools. However, some filnk-native users may prefer to use the
> REST
> > >> API.
> > >> > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > >> > > >
> > >> > > > In the FLIP, we also propose the REST endpoint, which has the
> > >> similar
> > >> > > > APIs compared to the gateway in the
> > ververica/flink-sql-gateway[2].
> > >> At
> > >> > > the
> > >> > > > last, we discuss how to use the SQL Client to submit the
> statement
> > >> to
> > >> > the
> > >> > > > Gateway with the REST API.
> > >> > > >
> > >> > > > I am glad that you can give some feedback about FLIP-91.
> > >> > > >
> > >> > > > Best,
> > >> > > > Shengkai
> > >> > > >
> > >> > > > [1]
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > >> > > > [2] https://github.com/ververica/flink-sql-gateway
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi Marijn and LuNing.

Thanks for your feedback!

> The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
which would be used by Flink's SQL Client (or other applications)?

Agreed. The FLIP mainly focus on the Gateway. I think it's better to rename
the name to the "Support SQL Gateway". WDYT?

> From a user perspective, I would have expected that we start with the
REST endpoint before explaining how we would integrate this into Flink. Now
it's quite hard to first understand what we want to offer to users and if
that will be sufficient for a first version.

emmm. Considering that api is basically the operation of some concepts, is
it better to introduce the core concepts first? But I agree you are right
that we should start with the RESt endpoint. I reorganize the content to
introduce the REST first in the public interfaces.

> With Flink 1.15, we're introducing an OpenAPI specification. Can we
also do this straight away for the REST Gateway?

Yes. We will organize the related APIs into OpenAPI specification.

>Should we introduce the REST Gateway as part of Flink's main repository?
>Wouldn't we be better off to maintain this in a separate repository under
>ASF?

I think it's better to intergate the Gateway into the Flink code base. The
reason behind is

1. The Gateway relies on the Flink implementation,  I think we'd better to
maintain it inside the Flink. It really takes us much time to upgrade the
sql-gateway in ververica repo to the latest Flink version.

2. The Gateway is important to the Flink itself. Many users needs the
Gateway to manage the Flink SQL jobs. Actually Hive, Spark both have its
Gateway in its code base.

But I think it's fine to put other utils, e.g. JDBC under the ASF.

> Ideally you would like to be able to support multiple Flink versions
> with one version of the REST Gateway I think?

> Users can upgrade a large number of Flink jobs versions gradually in a
Gateway service.

Because the Gateway itself relies on the Flink inner implementation...I
think we can just use one Gateway per versions. Users can manage the
gateway with other utils.

>There's no mention of Batch or Streaming in this concept. If I recall
>correctly, the current Flink SQL Gateway can only support Batch. How will
>we support Streaming?

> I can imagine that if a user wants to use a REST Gateway, there's also a
> strong need to combine this with a Catalog.

Yes. I add a section about the Usage of the Gateway. Users can use the SQL
do everything in the Gateway, including
- configure the execution parameter, including exectuion mode
- manage the metadata with DDL, e.g. register catalog
- submit the job
...

>Will there be any requirement with JDBC, as there currently is?

In the FLIP-223, we implement the HiveServer2 endpint. Users can use the
hive jdbc to connect to the Flink SQL Gateway.

> Shall we name this option `sql-gateway.session.init-file` and write it
into
the FLIP-91?

Actually we already supports the -i parameters in the sql client. What's
more, Hive also supports the -i parameter in the client side[1].
I think it's fine to move this functionlity to the client rather than
gateway. WDYT?

[1]
https://github.com/apache/hive/blob/c3fa88a1b7d1475f44383fca913aecf9c664bab0/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L321

Best,
Shengkai





LuNing Wang <wa...@gmail.com> 于2022年4月28日周四 10:04写道：

> > * Should we introduce the REST Gateway as part of Flink's main
> repository?
> Wouldn't we be better off to maintain this in a separate repository under
> ASF? Ideally you would like to be able to support multiple Flink versions
> with one version of the REST Gateway I think?
>
> We would be better off maintaining this in a separate repository. It is
> important to support multiple Flink versions. Users can upgrade a large
> number of Flink jobs versions gradually in a Gateway service.
>
> LuNing Wang <wa...@gmail.com> 于2022年4月27日周三 17:54写道：
>
> > Hi ShengKai,
> >
> > After I read FLIP-91[1], I want to add an init-file option. Its
> > functionality is the same as option '-i' of Flink SQL Client.
> >
> > When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by
> > this option after SQL Gateway starts every time.
> >
> > Shall we name this option `sql-gateway.session.init-file` and write it
> > into the FLIP-91?
> >
> > Best regards,
> >
> > LuNing Wang
> >
> > [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91
> >
> > Martijn Visser <ma...@ververica.com> 于2022年4月26日周二 20:32写道：
> >
> >> Hi Shengkai,
> >>
> >> Thanks for opening this discussion. I did a first brief pass over the
> FLIP
> >> and I have a couple of questions/remarks:
> >>
> >> * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
> >> which would be used by Flink's SQL Client (or other applications)?
> >>
> >> * From a user perspective, I would have expected that we start with the
> >> REST endpoint before explaining how we would integrate this into Flink.
> >> Now
> >> it's quite hard to first understand what we want to offer to users and
> if
> >> that will be sufficient for a first version.
> >>
> >> * With Flink 1.15, we're introducing an OpenAPI specification [1]. Can
> we
> >> also do this straight away for the REST Gateway?
> >>
> >> * Should we introduce the REST Gateway as part of Flink's main
> repository?
> >> Wouldn't we be better off to maintain this in a separate repository
> under
> >> ASF? Ideally you would like to be able to support multiple Flink
> versions
> >> with one version of the REST Gateway I think?
> >>
> >> * There's no mention of Batch or Streaming in this concept. If I recall
> >> correctly, the current Flink SQL Gateway can only support Batch. How
> will
> >> we support Streaming? Will there be any requirement with JDBC, as there
> >> currently is?
> >>
> >> * I can imagine that if a user wants to use a REST Gateway, there's
> also a
> >> strong need to combine this with a Catalog. Do you think this should be
> >> part of this FLIP?
> >>
> >> Best regards,
> >>
> >> Martijn Visser
> >> https://twitter.com/MartijnVisser82
> >> https://github.com/MartijnVisser
> >>
> >> [1]
> >>
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager
> >>
> >> On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fs...@gmail.com> wrote:
> >>
> >> > Hi. Jiang.
> >> >
> >> > Thanks for your feedback！
> >> >
> >> > > Do the public interfaces of GatewayService refer to any service?
> >> >
> >> > We will only expose one GatewayService implementation. We will put the
> >> > interface into the common package and the developer who wants to
> >> implement
> >> > a new endpoint can just rely on the interface package rather than the
> >> > implementation.
> >> >
> >> > > What's the behavior of SQL Client Gateway working on Yarn or K8S?
> Does
> >> > the SQL Client Gateway support application or session mode on Yarn?
> >> >
> >> > I think we can support SQL Client Gateway to submit the jobs in
> >> > application/sesison mode.
> >> >
> >> > > Is there any event trigger in the operation state machine?
> >> >
> >> > Yes. I have already updated the content and add more details about the
> >> > state machine. During the revise, I found that I mix up the two
> >> concepts:
> >> > job submission and job execution. In fact, we only control the
> >> submission
> >> > mode at the gateway layer. Therefore, we don't need to mapping the
> >> > JobStatus here. If the user expects that the synchronization behavior
> >> is to
> >> > wait for the completion of the job execution before allowing the next
> >> > statement to be executed, then the Operation lifecycle should also
> >> contains
> >> > the job's execution, which means users should set `table.dml-sync`.
> >> >
> >> > > What's the return schema for the public interfaces of
> GatewayService?
> >> > Like getTable interface, what's the return value schema?
> >> >
> >> > The API of the GatewayService return the java objects and the endpoint
> >> can
> >> > organize the objects with expected schema. The return results is also
> >> list
> >> > the section ComponetAPI#GatewayService#API. The return type of the
> >> > GatewayService#getTable is `ContextResolvedTable`.
> >> >
> >> > > How does the user get the operation log?
> >> >
> >> > The OperationManager will register the LogAppender before the
> Operation
> >> > execution. The Log Appender will hijack the logger and also write the
> >> log
> >> > that related to the Operation to another files. When users wants to
> >> fetch
> >> > the Operation log, the GatewayService will read the content in the
> file
> >> and
> >> > return.
> >> >
> >> > Best,
> >> > Shengkai
> >> >
> >> >
> >> >
> >> >
> >> > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> >> >
> >> > > Hi Shengkai.
> >> > >
> >> > > Thanks for driving the proposal of SQL Client Gateway. I have some
> >> > > knowledge of Kyuubi and have some questions about the design:
> >> > >
> >> > > 1.Do the public interfaces of GatewayService refer to any service?
> If
> >> > > referring to HiveService, does GatewayService need interfaces like
> >> > > getQueryId etc.
> >> > >
> >> > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S?
> >> Does
> >> > > the SQL Client Gateway support application or session mode on Yarn?
> >> > >
> >> > > 3.Is there any event trigger in the operation state machine?
> >> > >
> >> > > 4.What's the return schema for the public interfaces of
> >> GatewayService?
> >> > > Like getTable interface, what's the return value schema?
> >> > >
> >> > > 5.How does the user get the operation log?
> >> > >
> >> > > Thanks,
> >> > > Nicholas Jiang
> >> > >
> >> > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> >> > > > Hi, Flink developers.
> >> > > >
> >> > > > I want to start a discussion about the FLIP-91: Support Flink SQL
> >> > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
> >> submit
> >> > > and
> >> > > > manage their jobs in the online environment with the pluggable
> >> > endpoints.
> >> > > > The reason why we introduce the Gateway with pluggable endpoints
> is
> >> > that
> >> > > > many users have their preferences. For example, the HiveServer2
> >> users
> >> > > > prefer to use the gateway with HiveServer2-style API, which has
> >> > numerous
> >> > > > tools. However, some filnk-native users may prefer to use the REST
> >> API.
> >> > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> >> > > >
> >> > > > In the FLIP, we also propose the REST endpoint, which has the
> >> similar
> >> > > > APIs compared to the gateway in the
> ververica/flink-sql-gateway[2].
> >> At
> >> > > the
> >> > > > last, we discuss how to use the SQL Client to submit the statement
> >> to
> >> > the
> >> > > > Gateway with the REST API.
> >> > > >
> >> > > > I am glad that you can give some feedback about FLIP-91.
> >> > > >
> >> > > > Best,
> >> > > > Shengkai
> >> > > >
> >> > > > [1]
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> >> > > > [2] https://github.com/ververica/flink-sql-gateway
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[LuNing Wang <wa...@gmail.com>]

> * Should we introduce the REST Gateway as part of Flink's main repository?
Wouldn't we be better off to maintain this in a separate repository under
ASF? Ideally you would like to be able to support multiple Flink versions
with one version of the REST Gateway I think?

We would be better off maintaining this in a separate repository. It is
important to support multiple Flink versions. Users can upgrade a large
number of Flink jobs versions gradually in a Gateway service.

LuNing Wang <wa...@gmail.com> 于2022年4月27日周三 17:54写道：

> Hi ShengKai,
>
> After I read FLIP-91[1], I want to add an init-file option. Its
> functionality is the same as option '-i' of Flink SQL Client.
>
> When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by
> this option after SQL Gateway starts every time.
>
> Shall we name this option `sql-gateway.session.init-file` and write it
> into the FLIP-91?
>
> Best regards,
>
> LuNing Wang
>
> [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91
>
> Martijn Visser <ma...@ververica.com> 于2022年4月26日周二 20:32写道：
>
>> Hi Shengkai,
>>
>> Thanks for opening this discussion. I did a first brief pass over the FLIP
>> and I have a couple of questions/remarks:
>>
>> * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
>> which would be used by Flink's SQL Client (or other applications)?
>>
>> * From a user perspective, I would have expected that we start with the
>> REST endpoint before explaining how we would integrate this into Flink.
>> Now
>> it's quite hard to first understand what we want to offer to users and if
>> that will be sufficient for a first version.
>>
>> * With Flink 1.15, we're introducing an OpenAPI specification [1]. Can we
>> also do this straight away for the REST Gateway?
>>
>> * Should we introduce the REST Gateway as part of Flink's main repository?
>> Wouldn't we be better off to maintain this in a separate repository under
>> ASF? Ideally you would like to be able to support multiple Flink versions
>> with one version of the REST Gateway I think?
>>
>> * There's no mention of Batch or Streaming in this concept. If I recall
>> correctly, the current Flink SQL Gateway can only support Batch. How will
>> we support Streaming? Will there be any requirement with JDBC, as there
>> currently is?
>>
>> * I can imagine that if a user wants to use a REST Gateway, there's also a
>> strong need to combine this with a Catalog. Do you think this should be
>> part of this FLIP?
>>
>> Best regards,
>>
>> Martijn Visser
>> https://twitter.com/MartijnVisser82
>> https://github.com/MartijnVisser
>>
>> [1]
>>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager
>>
>> On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fs...@gmail.com> wrote:
>>
>> > Hi. Jiang.
>> >
>> > Thanks for your feedback！
>> >
>> > > Do the public interfaces of GatewayService refer to any service?
>> >
>> > We will only expose one GatewayService implementation. We will put the
>> > interface into the common package and the developer who wants to
>> implement
>> > a new endpoint can just rely on the interface package rather than the
>> > implementation.
>> >
>> > > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
>> > the SQL Client Gateway support application or session mode on Yarn?
>> >
>> > I think we can support SQL Client Gateway to submit the jobs in
>> > application/sesison mode.
>> >
>> > > Is there any event trigger in the operation state machine?
>> >
>> > Yes. I have already updated the content and add more details about the
>> > state machine. During the revise, I found that I mix up the two
>> concepts:
>> > job submission and job execution. In fact, we only control the
>> submission
>> > mode at the gateway layer. Therefore, we don't need to mapping the
>> > JobStatus here. If the user expects that the synchronization behavior
>> is to
>> > wait for the completion of the job execution before allowing the next
>> > statement to be executed, then the Operation lifecycle should also
>> contains
>> > the job's execution, which means users should set `table.dml-sync`.
>> >
>> > > What's the return schema for the public interfaces of GatewayService?
>> > Like getTable interface, what's the return value schema?
>> >
>> > The API of the GatewayService return the java objects and the endpoint
>> can
>> > organize the objects with expected schema. The return results is also
>> list
>> > the section ComponetAPI#GatewayService#API. The return type of the
>> > GatewayService#getTable is `ContextResolvedTable`.
>> >
>> > > How does the user get the operation log?
>> >
>> > The OperationManager will register the LogAppender before the Operation
>> > execution. The Log Appender will hijack the logger and also write the
>> log
>> > that related to the Operation to another files. When users wants to
>> fetch
>> > the Operation log, the GatewayService will read the content in the file
>> and
>> > return.
>> >
>> > Best,
>> > Shengkai
>> >
>> >
>> >
>> >
>> > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
>> >
>> > > Hi Shengkai.
>> > >
>> > > Thanks for driving the proposal of SQL Client Gateway. I have some
>> > > knowledge of Kyuubi and have some questions about the design:
>> > >
>> > > 1.Do the public interfaces of GatewayService refer to any service? If
>> > > referring to HiveService, does GatewayService need interfaces like
>> > > getQueryId etc.
>> > >
>> > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S?
>> Does
>> > > the SQL Client Gateway support application or session mode on Yarn?
>> > >
>> > > 3.Is there any event trigger in the operation state machine?
>> > >
>> > > 4.What's the return schema for the public interfaces of
>> GatewayService?
>> > > Like getTable interface, what's the return value schema?
>> > >
>> > > 5.How does the user get the operation log?
>> > >
>> > > Thanks,
>> > > Nicholas Jiang
>> > >
>> > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
>> > > > Hi, Flink developers.
>> > > >
>> > > > I want to start a discussion about the FLIP-91: Support Flink SQL
>> > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
>> submit
>> > > and
>> > > > manage their jobs in the online environment with the pluggable
>> > endpoints.
>> > > > The reason why we introduce the Gateway with pluggable endpoints is
>> > that
>> > > > many users have their preferences. For example, the HiveServer2
>> users
>> > > > prefer to use the gateway with HiveServer2-style API, which has
>> > numerous
>> > > > tools. However, some filnk-native users may prefer to use the REST
>> API.
>> > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
>> > > >
>> > > > In the FLIP, we also propose the REST endpoint, which has the
>> similar
>> > > > APIs compared to the gateway in the ververica/flink-sql-gateway[2].
>> At
>> > > the
>> > > > last, we discuss how to use the SQL Client to submit the statement
>> to
>> > the
>> > > > Gateway with the REST API.
>> > > >
>> > > > I am glad that you can give some feedback about FLIP-91.
>> > > >
>> > > > Best,
>> > > > Shengkai
>> > > >
>> > > > [1]
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
>> > > > [2] https://github.com/ververica/flink-sql-gateway
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[LuNing Wang <wa...@gmail.com>]

Hi ShengKai,

After I read FLIP-91[1], I want to add an init-file option. Its
functionality is the same as option '-i' of Flink SQL Client.

When I use Catalog(HiveCatalog), I need to execute `CREATE CATALOG` by this
option after SQL Gateway starts every time.

Shall we name this option `sql-gateway.session.init-file` and write it into
the FLIP-91?

Best regards,

LuNing Wang

[1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-91

Martijn Visser <ma...@ververica.com> 于2022年4月26日周二 20:32写道：

> Hi Shengkai,
>
> Thanks for opening this discussion. I did a first brief pass over the FLIP
> and I have a couple of questions/remarks:
>
> * The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
> which would be used by Flink's SQL Client (or other applications)?
>
> * From a user perspective, I would have expected that we start with the
> REST endpoint before explaining how we would integrate this into Flink. Now
> it's quite hard to first understand what we want to offer to users and if
> that will be sufficient for a first version.
>
> * With Flink 1.15, we're introducing an OpenAPI specification [1]. Can we
> also do this straight away for the REST Gateway?
>
> * Should we introduce the REST Gateway as part of Flink's main repository?
> Wouldn't we be better off to maintain this in a separate repository under
> ASF? Ideally you would like to be able to support multiple Flink versions
> with one version of the REST Gateway I think?
>
> * There's no mention of Batch or Streaming in this concept. If I recall
> correctly, the current Flink SQL Gateway can only support Batch. How will
> we support Streaming? Will there be any requirement with JDBC, as there
> currently is?
>
> * I can imagine that if a user wants to use a REST Gateway, there's also a
> strong need to combine this with a Catalog. Do you think this should be
> part of this FLIP?
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager
>
> On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fs...@gmail.com> wrote:
>
> > Hi. Jiang.
> >
> > Thanks for your feedback！
> >
> > > Do the public interfaces of GatewayService refer to any service?
> >
> > We will only expose one GatewayService implementation. We will put the
> > interface into the common package and the developer who wants to
> implement
> > a new endpoint can just rely on the interface package rather than the
> > implementation.
> >
> > > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> > the SQL Client Gateway support application or session mode on Yarn?
> >
> > I think we can support SQL Client Gateway to submit the jobs in
> > application/sesison mode.
> >
> > > Is there any event trigger in the operation state machine?
> >
> > Yes. I have already updated the content and add more details about the
> > state machine. During the revise, I found that I mix up the two concepts:
> > job submission and job execution. In fact, we only control the submission
> > mode at the gateway layer. Therefore, we don't need to mapping the
> > JobStatus here. If the user expects that the synchronization behavior is
> to
> > wait for the completion of the job execution before allowing the next
> > statement to be executed, then the Operation lifecycle should also
> contains
> > the job's execution, which means users should set `table.dml-sync`.
> >
> > > What's the return schema for the public interfaces of GatewayService?
> > Like getTable interface, what's the return value schema?
> >
> > The API of the GatewayService return the java objects and the endpoint
> can
> > organize the objects with expected schema. The return results is also
> list
> > the section ComponetAPI#GatewayService#API. The return type of the
> > GatewayService#getTable is `ContextResolvedTable`.
> >
> > > How does the user get the operation log?
> >
> > The OperationManager will register the LogAppender before the Operation
> > execution. The Log Appender will hijack the logger and also write the log
> > that related to the Operation to another files. When users wants to fetch
> > the Operation log, the GatewayService will read the content in the file
> and
> > return.
> >
> > Best,
> > Shengkai
> >
> >
> >
> >
> > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
> >
> > > Hi Shengkai.
> > >
> > > Thanks for driving the proposal of SQL Client Gateway. I have some
> > > knowledge of Kyuubi and have some questions about the design:
> > >
> > > 1.Do the public interfaces of GatewayService refer to any service? If
> > > referring to HiveService, does GatewayService need interfaces like
> > > getQueryId etc.
> > >
> > > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S?
> Does
> > > the SQL Client Gateway support application or session mode on Yarn?
> > >
> > > 3.Is there any event trigger in the operation state machine?
> > >
> > > 4.What's the return schema for the public interfaces of GatewayService?
> > > Like getTable interface, what's the return value schema?
> > >
> > > 5.How does the user get the operation log?
> > >
> > > Thanks,
> > > Nicholas Jiang
> > >
> > > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > > Hi, Flink developers.
> > > >
> > > > I want to start a discussion about the FLIP-91: Support Flink SQL
> > > > Gateway[1]. Flink SQL Gateway is a service that allows users to
> submit
> > > and
> > > > manage their jobs in the online environment with the pluggable
> > endpoints.
> > > > The reason why we introduce the Gateway with pluggable endpoints is
> > that
> > > > many users have their preferences. For example, the HiveServer2 users
> > > > prefer to use the gateway with HiveServer2-style API, which has
> > numerous
> > > > tools. However, some filnk-native users may prefer to use the REST
> API.
> > > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > > >
> > > > In the FLIP, we also propose the REST endpoint, which has the similar
> > > > APIs compared to the gateway in the ververica/flink-sql-gateway[2].
> At
> > > the
> > > > last, we discuss how to use the SQL Client to submit the statement to
> > the
> > > > Gateway with the REST API.
> > > >
> > > > I am glad that you can give some feedback about FLIP-91.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > > [2] https://github.com/ververica/flink-sql-gateway
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Martijn Visser <ma...@ververica.com>]

Hi Shengkai,

Thanks for opening this discussion. I did a first brief pass over the FLIP
and I have a couple of questions/remarks:

* The FLIP is called "SQL Client Gateway", but isn't this a REST Gateway
which would be used by Flink's SQL Client (or other applications)?

* From a user perspective, I would have expected that we start with the
REST endpoint before explaining how we would integrate this into Flink. Now
it's quite hard to first understand what we want to offer to users and if
that will be sufficient for a first version.

* With Flink 1.15, we're introducing an OpenAPI specification [1]. Can we
also do this straight away for the REST Gateway?

* Should we introduce the REST Gateway as part of Flink's main repository?
Wouldn't we be better off to maintain this in a separate repository under
ASF? Ideally you would like to be able to support multiple Flink versions
with one version of the REST Gateway I think?

* There's no mention of Batch or Streaming in this concept. If I recall
correctly, the current Flink SQL Gateway can only support Batch. How will
we support Streaming? Will there be any requirement with JDBC, as there
currently is?

* I can imagine that if a user wants to use a REST Gateway, there's also a
strong need to combine this with a Catalog. Do you think this should be
part of this FLIP?

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobmanager

On Sun, 24 Apr 2022 at 05:29, Shengkai Fang <fs...@gmail.com> wrote:

> Hi. Jiang.
>
> Thanks for your feedback！
>
> > Do the public interfaces of GatewayService refer to any service?
>
> We will only expose one GatewayService implementation. We will put the
> interface into the common package and the developer who wants to implement
> a new endpoint can just rely on the interface package rather than the
> implementation.
>
> > What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> the SQL Client Gateway support application or session mode on Yarn?
>
> I think we can support SQL Client Gateway to submit the jobs in
> application/sesison mode.
>
> > Is there any event trigger in the operation state machine?
>
> Yes. I have already updated the content and add more details about the
> state machine. During the revise, I found that I mix up the two concepts:
> job submission and job execution. In fact, we only control the submission
> mode at the gateway layer. Therefore, we don't need to mapping the
> JobStatus here. If the user expects that the synchronization behavior is to
> wait for the completion of the job execution before allowing the next
> statement to be executed, then the Operation lifecycle should also contains
> the job's execution, which means users should set `table.dml-sync`.
>
> > What's the return schema for the public interfaces of GatewayService?
> Like getTable interface, what's the return value schema?
>
> The API of the GatewayService return the java objects and the endpoint can
> organize the objects with expected schema. The return results is also list
> the section ComponetAPI#GatewayService#API. The return type of the
> GatewayService#getTable is `ContextResolvedTable`.
>
> > How does the user get the operation log?
>
> The OperationManager will register the LogAppender before the Operation
> execution. The Log Appender will hijack the logger and also write the log
> that related to the Operation to another files. When users wants to fetch
> the Operation log, the GatewayService will read the content in the file and
> return.
>
> Best,
> Shengkai
>
>
>
>
> Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：
>
> > Hi Shengkai.
> >
> > Thanks for driving the proposal of SQL Client Gateway. I have some
> > knowledge of Kyuubi and have some questions about the design:
> >
> > 1.Do the public interfaces of GatewayService refer to any service? If
> > referring to HiveService, does GatewayService need interfaces like
> > getQueryId etc.
> >
> > 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> > the SQL Client Gateway support application or session mode on Yarn?
> >
> > 3.Is there any event trigger in the operation state machine?
> >
> > 4.What's the return schema for the public interfaces of GatewayService?
> > Like getTable interface, what's the return value schema?
> >
> > 5.How does the user get the operation log?
> >
> > Thanks,
> > Nicholas Jiang
> >
> > On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > > Hi, Flink developers.
> > >
> > > I want to start a discussion about the FLIP-91: Support Flink SQL
> > > Gateway[1]. Flink SQL Gateway is a service that allows users to submit
> > and
> > > manage their jobs in the online environment with the pluggable
> endpoints.
> > > The reason why we introduce the Gateway with pluggable endpoints is
> that
> > > many users have their preferences. For example, the HiveServer2 users
> > > prefer to use the gateway with HiveServer2-style API, which has
> numerous
> > > tools. However, some filnk-native users may prefer to use the REST API.
> > > Therefore, we propose the SQL Gateway with pluggable endpoint.
> > >
> > > In the FLIP, we also propose the REST endpoint, which has the similar
> > > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At
> > the
> > > last, we discuss how to use the SQL Client to submit the statement to
> the
> > > Gateway with the REST API.
> > >
> > > I am glad that you can give some feedback about FLIP-91.
> > >
> > > Best,
> > > Shengkai
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > > [2] https://github.com/ververica/flink-sql-gateway
> > >
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang <fs...@gmail.com>]

Hi. Jiang.

Thanks for your feedback！

> Do the public interfaces of GatewayService refer to any service?

We will only expose one GatewayService implementation. We will put the
interface into the common package and the developer who wants to implement
a new endpoint can just rely on the interface package rather than the
implementation.

> What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
the SQL Client Gateway support application or session mode on Yarn?

I think we can support SQL Client Gateway to submit the jobs in
application/sesison mode.

> Is there any event trigger in the operation state machine?

Yes. I have already updated the content and add more details about the
state machine. During the revise, I found that I mix up the two concepts:
job submission and job execution. In fact, we only control the submission
mode at the gateway layer. Therefore, we don't need to mapping the
JobStatus here. If the user expects that the synchronization behavior is to
wait for the completion of the job execution before allowing the next
statement to be executed, then the Operation lifecycle should also contains
the job's execution, which means users should set `table.dml-sync`.

> What's the return schema for the public interfaces of GatewayService?
Like getTable interface, what's the return value schema?

The API of the GatewayService return the java objects and the endpoint can
organize the objects with expected schema. The return results is also list
the section ComponetAPI#GatewayService#API. The return type of the
GatewayService#getTable is `ContextResolvedTable`.

> How does the user get the operation log?

The OperationManager will register the LogAppender before the Operation
execution. The Log Appender will hijack the logger and also write the log
that related to the Operation to another files. When users wants to fetch
the Operation log, the GatewayService will read the content in the file and
return.

Best,
Shengkai




Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:21写道：

> Hi Shengkai.
>
> Thanks for driving the proposal of SQL Client Gateway. I have some
> knowledge of Kyuubi and have some questions about the design:
>
> 1.Do the public interfaces of GatewayService refer to any service? If
> referring to HiveService, does GatewayService need interfaces like
> getQueryId etc.
>
> 2.What's the behavior of SQL Client Gateway working on Yarn or K8S? Does
> the SQL Client Gateway support application or session mode on Yarn?
>
> 3.Is there any event trigger in the operation state machine?
>
> 4.What's the return schema for the public interfaces of GatewayService?
> Like getTable interface, what's the return value schema?
>
> 5.How does the user get the operation log?
>
> Thanks,
> Nicholas Jiang
>
> On 2022/04/21 06:42:30 Shengkai Fang wrote:
> > Hi, Flink developers.
> >
> > I want to start a discussion about the FLIP-91: Support Flink SQL
> > Gateway[1]. Flink SQL Gateway is a service that allows users to submit
> and
> > manage their jobs in the online environment with the pluggable endpoints.
> > The reason why we introduce the Gateway with pluggable endpoints is that
> > many users have their preferences. For example, the HiveServer2 users
> > prefer to use the gateway with HiveServer2-style API, which has numerous
> > tools. However, some filnk-native users may prefer to use the REST API.
> > Therefore, we propose the SQL Gateway with pluggable endpoint.
> >
> > In the FLIP, we also propose the REST endpoint, which has the similar
> > APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At
> the
> > last, we discuss how to use the SQL Client to submit the statement to the
> > Gateway with the REST API.
> >
> > I am glad that you can give some feedback about FLIP-91.
> >
> > Best,
> > Shengkai
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> > [2] https://github.com/ververica/flink-sql-gateway
> >
>

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Nicholas Jiang <ni...@apache.org>]

Hi Shengkai.

Thanks for driving the proposal of SQL Client Gateway. I have some knowledge of Kyuubi and have some questions about the design:

1.Do the public interfaces of GatewayService refer to any service? If referring to HiveService, does GatewayService need interfaces like getQueryId etc.

2.What's the behavior of SQL Client Gateway working on Yarn or K8S? Does the SQL Client Gateway support application or session mode on Yarn?

3.Is there any event trigger in the operation state machine?

4.What's the return schema for the public interfaces of GatewayService? Like getTable interface, what's the return value schema?

5.How does the user get the operation log?

Thanks,
Nicholas Jiang

On 2022/04/21 06:42:30 Shengkai Fang wrote:
> Hi, Flink developers.
> 
> I want to start a discussion about the FLIP-91: Support Flink SQL
> Gateway[1]. Flink SQL Gateway is a service that allows users to submit and
> manage their jobs in the online environment with the pluggable endpoints.
> The reason why we introduce the Gateway with pluggable endpoints is that
> many users have their preferences. For example, the HiveServer2 users
> prefer to use the gateway with HiveServer2-style API, which has numerous
> tools. However, some filnk-native users may prefer to use the REST API.
> Therefore, we propose the SQL Gateway with pluggable endpoint.
> 
> In the FLIP, we also propose the REST endpoint, which has the similar
> APIs compared to the gateway in the ververica/flink-sql-gateway[2]. At the
> last, we discuss how to use the SQL Client to submit the statement to the
> Gateway with the REST API.
> 
> I am glad that you can give some feedback about FLIP-91.
> 
> Best,
> Shengkai
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-91%3A+Support+SQL+Client+Gateway
> [2] https://github.com/ververica/flink-sql-gateway
>