You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Martijn Visser <ma...@apache.org> on 2022/05/04 13:44:27 UTC
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Hi Shengkai,
> Actually we will only rely on the API in the Hive, which only contains
the thrift file and the generated code
So this implementation would not rely in any way on Hive, only on Thrift?
Best regards,
Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser
On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com> wrote:
> Hi, Jark and Martijn
>
> Thanks for your feedback.
>
> > Kyuubi provides three ways to configure Hive metastore [1]. Could we
> provide similar abilities?
>
> Yes. I have updated the FLIP about this and it takes some time to figure
> out how the jdbc driver works. I added the section about how to use the
> hive JDBC to configure the session-level catalog.
>
> > I think we can improve the "HiveServer2 Compatibility" section.
>
> Yes. I have updated the FLIP and added more details about the
> compatibility.
>
> > Prefer to first complete the discussion and vote on FLIP-91 then discuss
> FLIP-223
>
> Of course. We can wait until the discussion of the FLIP-91 finishes.
>
> > Maintenance concerns about the hive
>
> Actually we will only rely on the API in the Hive, which only contains the
> thrift file and the generated code[1]. I think it will not influence us to
> upgrade the java version.
>
> [1] https://github.com/apache/hive/tree/master/service-rpc
>
> Best,
> Shengkai
>
> Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
>
> > Hi all,
> >
> > I'm not too familiar with Hive and HiveServer2, but I do have a couple of
> > questions/concerns:
> >
> > 1. What is the relationship between this FLIP and FLIP-91? My assumption
> > would be that this FLIP (and therefore the HiveServer2) implementation
> > would need to be integrated in the REST Gateway, is that correct? If so,
> I
> > would prefer to first complete the discussion and vote on FLIP-91, else
> > we'll have two moving FLIPs who have a direct relationship with each
> other.
> >
> > 2. While I understand that Hive is important (in the Chinese ecosystem,
> not
> > so much in Europe and the US), I still have maintenance concerns on this
> > topic. We know that the current Hive integration isn't exactly ideal and
> > requires a lot of work to get in better shape. At the same time, Hive
> still
> > doesn't support Java 11 while we need (and should, given the premier
> > support has ended already) to move away from Java 8.
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> >
> > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com> wrote:
> >
> > > Thank Shengkai for driving this effort,
> > > I think this is an essential addition to Flink Batch.
> > >
> > > I have some small suggestions:
> > > 1) Kyuubi provides three ways to configure Hive metastore [1]. Could we
> > > provide similar abilities?
> > > Especially with the JDBC Connection URL, users can visit different Hive
> > > metastore server instances.
> > >
> > > 2) I think we can improve the "HiveServer2 Compatibility" section.
> > > We need to figure out two compatibility matrices. One is SQL Gateway
> with
> > > different versions of Hive metastore,
> > > and the other is different versions of Hive client (e.g., Hive JDBC)
> with
> > > SQL Gateway. We need to clarify
> > > what metastore and client versions we support and how users configure
> the
> > > versions.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > [1]:
> > >
> > >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > >
> > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fs...@gmail.com> wrote:
> > >
> > > > Hi, Jiang.
> > > >
> > > > Thanks for your feedback!
> > > >
> > > > > Integrating the Hive ecosystem should not require changing the
> > service
> > > > interface
> > > >
> > > > I move the API change to the FLIP-91. But I think it's possible we
> add
> > > more
> > > > interfaces to intergrate the new endpoints in the future because
> every
> > > > endpoints's functionality is different. For example, the REST
> endpoint
> > > > doen't support to fetch operation-level logs but the hiveserver2
> > endpoint
> > > > supports. In this case, we need to modify the shared GatewayService
> to
> > > > support the functionality exposed by the new endpint.
> > > >
> > > > > How to support different Hive versions?
> > > >
> > > > Do you means to support the different HiveServer2 version? The
> > > HiveServer2
> > > > uses the version to guarantee the compatibility. During the
> > openSession,
> > > > the client and server will determine the protocol
> > version(minimun(client
> > > > version, hiveendpoint version)). After that the client and the server
> > > uses
> > > > the determined version to communicate. In the HiveServer2 endpoint,
> it
> > > > determines how the endpoint deserialize the results and the result
> > > schema.
> > > > I add a section about HiveServer2 compatiblity.
> > > >
> > > > > Could you please fully provide its definition including input
> > > parameters
> > > > and the corresponding return value schema?
> > > >
> > > > Because we implements the interface exposed by the Hive. So I add the
> > > file
> > > > link to the HiveServer2 interfaces[1], which contains all input
> > > parameters
> > > > and the results. Considering the file doesn't contain the output for
> > the
> > > > Operation, I add the output schema for all the supported Operation in
> > the
> > > > FLIP, which is not covered in the link. Hope these can address your
> > > > question.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:43写道：
> > > >
> > > > > Hi Shengkai.
> > > > >
> > > > > Thanks for driving the proposal of HiveServer2 Endpoint support.
> For
> > > the
> > > > > "GatewayService API Change", I don't think the motivation for
> > > supporting
> > > > > HiveServer2 endpoint need to change the GatewayService API, in
> other
> > > > words,
> > > > > integrating the Hive ecosystem should not require changing the
> > service
> > > > > interface. If you confirm to change GatewayService interface, IMO,
> > the
> > > > > proposal could be discussed in FLIP-91 because the public
> interfaces
> > > are
> > > > > defined in FLIP-91.
> > > > >
> > > > > In addtion, how to support different Hive versions and how to
> > guarantee
> > > > > compatibility is not mentioned in the design. What's the behavior
> of
> > > the
> > > > > compatibility?
> > > > >
> > > > > Finally, for the public interfaces, could you please fully provide
> > its
> > > > > definition including input parameters and the corresponding return
> > > value
> > > > > schema?
> > > > >
> > > > > Thanks,
> > > > > Nicholas Jiang
> > > > >
> > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > Hi, Flink developers.
> > > > > >
> > > > > > I want to start a discussion about the FLIP-223: Support
> > HiveServer2
> > > > > > Endpoint[1]. The Endpoint will implement the thrift interface
> > exposed
> > > > by
> > > > > > the HiveServer2, and users' BI, CLI and other tools based on the
> > > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL
> > Gateway.
> > > > > After
> > > > > > the FLIP finishes, the users can have almost the same experience
> in
> > > the
> > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the
> > > HiveServer2.
> > > > > >
> > > > > >
> > > > > > I am glad that you can give some feedback about FLIP-223.
> > > > > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Shengkai Fang <fs...@gmail.com>.
>
> Hi, everyone.
>
> Thanks for all the input. Hope my feedback can address most of the
> questions.
>
> Since the discussion has lasted for a long time, I think we have reached a
> consensus about the
> design. If the discussion doesn't get more responses, I think we can start
> the vote tomorrow.
>
> Best,
> Shengkai
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by godfrey he <go...@gmail.com>.
Hi, Shengkai.
Thanks for the update, LGTM now.
Best,
Godfrey
Shengkai Fang <fs...@gmail.com> 于2022年6月6日周一 16:47写道：
>
> Hi. Godfrey.
>
> Nice to hear the comments from you.
>
> > Could you give a whole architecture about the Ecosystem of HiveServers
> > and the SqlGateway, such as JDBC driver, Beeline, etc.
> > Which is more clear for users.
>
> Yes. I have updated the FLIP and added the architecture of the Gateway with
> the HiveServer2 endpoint.
>
> > How To Use
> >> Could you give a complete example to describe an end-to-end case?
>
> Yes. I have updated the FLIP. The beeline users can just use the connect
> command to connect to the SQLGateway with the HiveServer2 endpoint.
> For example, users just inputs "!connect
> jdbc:hive2://<host>:<port>/<db>;auth=noSasl
> hiveuser pass" into the terminal to connect to the SQLGateway.
>
> > Is the streaming SQL supported? What's the behavior if I submit a
> streaming query or I change the dialect to 'default'?
> Yes. We don't limit the usage here. Users can switch to the streaming mode
> or use the default dialect. But we don't suggest users use the hive
> dialect in the streaming mode. As far as I know, it has some problems that
> are not fixed yet, e.g. you may get errors for SQL that works in the batch
> mode. I added a section to mention this.
>
> > Considering the different users may have different requirements to
> connect to different meta stores,
> > they can use the DDL to register the HiveCatalog that satisfies their
> requirements.
> >> Could you give some examples to explain it more?
>
> Hive supports setting multiple metastore addresses via the config option
> "hive.metastore.urls". Here I just mean users can switch to connect to
> different metastore instances using the CREATE CATALOG DDL. I updated the
> FLIP to make it more clear.
>
> Best,
> Shengkai
>
> godfrey he <go...@gmail.com> 于2022年6月6日周一 13:45写道：
>
> > Hi Shengkai,
> >
> > Thanks for driving this.
> >
> > I have a few comments:
> >
> > Could you give a whole architecture about the Ecosystem of HiveServers
> > and the SqlGateway, such as JDBC driver, Beeline, etc.
> > Which is more clear for users.
> >
> > > Considering the different users may have different requirements to
> > connect to different meta stores,
> > > they can use the DDL to register the HiveCatalog that satisfies their
> > requirements.
> > Could you give some examples to explain it more?
> >
> > > How To Use
> > Could you a complete example to describe an end-to-end case?
> >
> > Is the streaming sql supported? What's the behavior if I submit streaming
> > query
> > or I change the dialect to 'default'?
> >
> > Best,
> > Godfrey
> >
> > Shengkai Fang <fs...@gmail.com> 于2022年6月1日周三 21:13写道：
> > >
> > > Hi, Jingsong.
> > >
> > > Thanks for your feedback.
> > >
> > > > I've read the FLIP and it's not quite clear what the specific
> > unsupported
> > > items are
> > >
> > > Yes. I have added a section named Difference with HiveServer2 and list
> > the
> > > difference between the SQL Gateway with HiveServer2 endpoint and
> > > HiveServer2.
> > >
> > > > Support multiple metastore clients in one gateway?
> > >
> > > Yes. It may cause class conflicts when using the different versions of
> > Hive
> > > Catalog at the same time. I add a section named "How to use" to remind
> > the
> > > users don't use HiveCatalog with different versions together.
> > >
> > > > Hive versions and setup
> > >
> > > Considering the HiveServer2 endpoint binds to the HiveCatalog, we will
> > not
> > > introduce a new module about the HiveServer2 endpoint. The current
> > > dependencies in the hive connector should be enough for the HiveServer2
> > > Endpoint except for the hive-service-RPC(it contains the HiveServer2
> > > interface). In this way, the hive connector jar will contain an
> > endpoint. I
> > > add a section named "Merge HiveServer2 Endpoint into Hive Connector
> > > Module".
> > >
> > > For usage, the user can just add the hive connector jar into the
> > classpath
> > > and use the sql-gateway.sh to start the SQL Gateway with the hiveserver2
> > > endpoint. You can refer to the section "How to use" for more details.
> > >
> > > Best,
> > > Shengkai
> > >
> > > Jingsong Li <ji...@gmail.com> 于2022年6月1日周三 15:04写道：
> > >
> > > > Hi Shengkai,
> > > >
> > > > Thanks for driving.
> > > >
> > > > I have a few comments:
> > > >
> > > > ## Unsupported features
> > > >
> > > > I've read the FLIP and it's not quite clear what the specific
> > unsupported
> > > > items are?
> > > > - For example, security related, is it not supported.
> > > > - For example, is there a loss of precision for types
> > > > - For example, the FetchResults are not the same
> > > >
> > > > ## Support multiple metastore clients in one gateway?
> > > >
> > > > > During the setup, the HiveServer2 tires to load the config in the
> > > > hive-site.xml to initialize the Hive metastore client. In the Flink,
> > we use
> > > > the Catalog interface to connect to the Hive Metastore, which is
> > allowed to
> > > > communicate with different Hive Metastore[1]. Therefore, we allows the
> > user
> > > > to specify the path of the hive-site.xml as the endpoint parameters,
> > which
> > > > will used to create the default HiveCatalog in the Flink. Considering
> > the
> > > > different users may have different requirements to connect to different
> > > > meta stores, they can use the DDL to register the HiveCatalog that
> > > > satisfies their requirements.
> > > >
> > > > I understand it is difficult. You really want to support?
> > > >
> > > > ## Hive versions and setup
> > > >
> > > > I saw jark also commented, but FLIP does not seem to have been
> > modified,
> > > > how should the user setup, which jar to add, which hive metastore
> > version
> > > > to support? How to setup to support?
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fs...@gmail.com>
> > wrote:
> > > >
> > > > > Hi, all.
> > > > >
> > > > > Considering we start to vote for FLIP-91 for a while, I think we can
> > > > > restart the discussion about the FLIP-223.
> > > > >
> > > > > I am glad that you can give some feedback about FLIP-223.
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > >
> > > > > Martijn Visser <ma...@ververica.com> 于2022年5月6日周五 19:10写道：
> > > > >
> > > > > > Hi Shengkai,
> > > > > >
> > > > > > Thanks for clarifying.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > Hi Martijn.
> > > > > > >
> > > > > > > > So this implementation would not rely in any way on Hive, only
> > on
> > > > > > Thrift?
> > > > > > >
> > > > > > > Yes. The dependency is light. We also can just copy the iface
> > file
> > > > > from
> > > > > > > the Hive repo and maintain by ourselves.
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
> > > > > > >
> > > > > > > Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> > > > > > >
> > > > > > > > Hi Shengkai,
> > > > > > > >
> > > > > > > > > Actually we will only rely on the API in the Hive, which only
> > > > > > contains
> > > > > > > > the thrift file and the generated code
> > > > > > > >
> > > > > > > > So this implementation would not rely in any way on Hive, only
> > on
> > > > > > Thrift?
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Martijn Visser
> > > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > > https://github.com/MartijnVisser
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fskmine@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Jark and Martijn
> > > > > > > > >
> > > > > > > > > Thanks for your feedback.
> > > > > > > > >
> > > > > > > > > > Kyuubi provides three ways to configure Hive metastore [1].
> > > > Could
> > > > > > we
> > > > > > > > > provide similar abilities?
> > > > > > > > >
> > > > > > > > > Yes. I have updated the FLIP about this and it takes some
> > time to
> > > > > > > figure
> > > > > > > > > out how the jdbc driver works. I added the section about how
> > to
> > > > use
> > > > > > the
> > > > > > > > > hive JDBC to configure the session-level catalog.
> > > > > > > > >
> > > > > > > > > > I think we can improve the "HiveServer2 Compatibility"
> > section.
> > > > > > > > >
> > > > > > > > > Yes. I have updated the FLIP and added more details about the
> > > > > > > > > compatibility.
> > > > > > > > >
> > > > > > > > > > Prefer to first complete the discussion and vote on
> > FLIP-91
> > > > then
> > > > > > > > discuss
> > > > > > > > > FLIP-223
> > > > > > > > >
> > > > > > > > > Of course. We can wait until the discussion of the FLIP-91
> > > > > finishes.
> > > > > > > > >
> > > > > > > > > > Maintenance concerns about the hive
> > > > > > > > >
> > > > > > > > > Actually we will only rely on the API in the Hive, which only
> > > > > > contains
> > > > > > > > the
> > > > > > > > > thrift file and the generated code[1]. I think it will not
> > > > > influence
> > > > > > us
> > > > > > > > to
> > > > > > > > > upgrade the java version.
> > > > > > > > >
> > > > > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shengkai
> > > > > > > > >
> > > > > > > > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二
> > 20:44写道：
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'm not too familiar with Hive and HiveServer2, but I do
> > have a
> > > > > > > couple
> > > > > > > > of
> > > > > > > > > > questions/concerns:
> > > > > > > > > >
> > > > > > > > > > 1. What is the relationship between this FLIP and FLIP-91?
> > My
> > > > > > > > assumption
> > > > > > > > > > would be that this FLIP (and therefore the HiveServer2)
> > > > > > > implementation
> > > > > > > > > > would need to be integrated in the REST Gateway, is that
> > > > correct?
> > > > > > If
> > > > > > > > so,
> > > > > > > > > I
> > > > > > > > > > would prefer to first complete the discussion and vote on
> > > > > FLIP-91,
> > > > > > > else
> > > > > > > > > > we'll have two moving FLIPs who have a direct relationship
> > with
> > > > > > each
> > > > > > > > > other.
> > > > > > > > > >
> > > > > > > > > > 2. While I understand that Hive is important (in the
> > Chinese
> > > > > > > ecosystem,
> > > > > > > > > not
> > > > > > > > > > so much in Europe and the US), I still have maintenance
> > > > concerns
> > > > > on
> > > > > > > > this
> > > > > > > > > > topic. We know that the current Hive integration isn't
> > exactly
> > > > > > ideal
> > > > > > > > and
> > > > > > > > > > requires a lot of work to get in better shape. At the same
> > > > time,
> > > > > > Hive
> > > > > > > > > still
> > > > > > > > > > doesn't support Java 11 while we need (and should, given
> > the
> > > > > > premier
> > > > > > > > > > support has ended already) to move away from Java 8.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > > Martijn Visser
> > > > > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > > > > https://github.com/MartijnVisser
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com>
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thank Shengkai for driving this effort,
> > > > > > > > > > > I think this is an essential addition to Flink Batch.
> > > > > > > > > > >
> > > > > > > > > > > I have some small suggestions:
> > > > > > > > > > > 1) Kyuubi provides three ways to configure Hive metastore
> > > > [1].
> > > > > > > Could
> > > > > > > > we
> > > > > > > > > > > provide similar abilities?
> > > > > > > > > > > Especially with the JDBC Connection URL, users can visit
> > > > > > different
> > > > > > > > Hive
> > > > > > > > > > > metastore server instances.
> > > > > > > > > > >
> > > > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility"
> > > > > > section.
> > > > > > > > > > > We need to figure out two compatibility matrices. One is
> > SQL
> > > > > > > Gateway
> > > > > > > > > with
> > > > > > > > > > > different versions of Hive metastore,
> > > > > > > > > > > and the other is different versions of Hive client (e.g.,
> > > > Hive
> > > > > > > JDBC)
> > > > > > > > > with
> > > > > > > > > > > SQL Gateway. We need to clarify
> > > > > > > > > > > what metastore and client versions we support and how
> > users
> > > > > > > configure
> > > > > > > > > the
> > > > > > > > > > > versions.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Jark
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > [1]:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > > > > > > > >
> > > > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <
> > > > fskmine@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Jiang.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your feedback!
> > > > > > > > > > > >
> > > > > > > > > > > > > Integrating the Hive ecosystem should not require
> > > > changing
> > > > > > the
> > > > > > > > > > service
> > > > > > > > > > > > interface
> > > > > > > > > > > >
> > > > > > > > > > > > I move the API change to the FLIP-91. But I think it's
> > > > > possible
> > > > > > > we
> > > > > > > > > add
> > > > > > > > > > > more
> > > > > > > > > > > > interfaces to intergrate the new endpoints in the
> > future
> > > > > > because
> > > > > > > > > every
> > > > > > > > > > > > endpoints's functionality is different. For example,
> > the
> > > > REST
> > > > > > > > > endpoint
> > > > > > > > > > > > doen't support to fetch operation-level logs but the
> > > > > > hiveserver2
> > > > > > > > > > endpoint
> > > > > > > > > > > > supports. In this case, we need to modify the shared
> > > > > > > GatewayService
> > > > > > > > > to
> > > > > > > > > > > > support the functionality exposed by the new endpint.
> > > > > > > > > > > >
> > > > > > > > > > > > > How to support different Hive versions?
> > > > > > > > > > > >
> > > > > > > > > > > > Do you means to support the different HiveServer2
> > version?
> > > > > The
> > > > > > > > > > > HiveServer2
> > > > > > > > > > > > uses the version to guarantee the compatibility.
> > During the
> > > > > > > > > > openSession,
> > > > > > > > > > > > the client and server will determine the protocol
> > > > > > > > > > version(minimun(client
> > > > > > > > > > > > version, hiveendpoint version)). After that the client
> > and
> > > > > the
> > > > > > > > server
> > > > > > > > > > > uses
> > > > > > > > > > > > the determined version to communicate. In the
> > HiveServer2
> > > > > > > endpoint,
> > > > > > > > > it
> > > > > > > > > > > > determines how the endpoint deserialize the results
> > and the
> > > > > > > result
> > > > > > > > > > > schema.
> > > > > > > > > > > > I add a section about HiveServer2 compatiblity.
> > > > > > > > > > > >
> > > > > > > > > > > > > Could you please fully provide its definition
> > including
> > > > > input
> > > > > > > > > > > parameters
> > > > > > > > > > > > and the corresponding return value schema?
> > > > > > > > > > > >
> > > > > > > > > > > > Because we implements the interface exposed by the
> > Hive.
> > > > So I
> > > > > > add
> > > > > > > > the
> > > > > > > > > > > file
> > > > > > > > > > > > link to the HiveServer2 interfaces[1], which contains
> > all
> > > > > input
> > > > > > > > > > > parameters
> > > > > > > > > > > > and the results. Considering the file doesn't contain
> > the
> > > > > > output
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > Operation, I add the output schema for all the
> > supported
> > > > > > > Operation
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > FLIP, which is not covered in the link. Hope these can
> > > > > address
> > > > > > > your
> > > > > > > > > > > > question.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Shengkai
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Nicholas Jiang <ni...@apache.org>
> > 于2022年4月22日周五
> > > > > > 16:43写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Shengkai.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for driving the proposal of HiveServer2
> > Endpoint
> > > > > > > support.
> > > > > > > > > For
> > > > > > > > > > > the
> > > > > > > > > > > > > "GatewayService API Change", I don't think the
> > motivation
> > > > > for
> > > > > > > > > > > supporting
> > > > > > > > > > > > > HiveServer2 endpoint need to change the
> > GatewayService
> > > > API,
> > > > > > in
> > > > > > > > > other
> > > > > > > > > > > > words,
> > > > > > > > > > > > > integrating the Hive ecosystem should not require
> > > > changing
> > > > > > the
> > > > > > > > > > service
> > > > > > > > > > > > > interface. If you confirm to change GatewayService
> > > > > interface,
> > > > > > > > IMO,
> > > > > > > > > > the
> > > > > > > > > > > > > proposal could be discussed in FLIP-91 because the
> > public
> > > > > > > > > interfaces
> > > > > > > > > > > are
> > > > > > > > > > > > > defined in FLIP-91.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In addtion, how to support different Hive versions
> > and
> > > > how
> > > > > to
> > > > > > > > > > guarantee
> > > > > > > > > > > > > compatibility is not mentioned in the design. What's
> > the
> > > > > > > behavior
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > compatibility?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Finally, for the public interfaces, could you please
> > > > fully
> > > > > > > > provide
> > > > > > > > > > its
> > > > > > > > > > > > > definition including input parameters and the
> > > > corresponding
> > > > > > > > return
> > > > > > > > > > > value
> > > > > > > > > > > > > schema?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Nicholas Jiang
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > > > > > > > Hi, Flink developers.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I want to start a discussion about the FLIP-223:
> > > > Support
> > > > > > > > > > HiveServer2
> > > > > > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift
> > > > > > interface
> > > > > > > > > > exposed
> > > > > > > > > > > > by
> > > > > > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools
> > > > based
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > > > > HiveServer2 can also be seamlessly migrated to the
> > > > Flink
> > > > > > SQL
> > > > > > > > > > Gateway.
> > > > > > > > > > > > > After
> > > > > > > > > > > > > > the FLIP finishes, the users can have almost the
> > same
> > > > > > > > experience
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as
> > in
> > > > the
> > > > > > > > > > > HiveServer2.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I am glad that you can give some feedback about
> > > > FLIP-223.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Shengkai
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Shengkai Fang <fs...@gmail.com>.
Hi. Godfrey.
Nice to hear the comments from you.
> Could you give a whole architecture about the Ecosystem of HiveServers
> and the SqlGateway, such as JDBC driver, Beeline, etc.
> Which is more clear for users.
Yes. I have updated the FLIP and added the architecture of the Gateway with
the HiveServer2 endpoint.
> How To Use
>> Could you give a complete example to describe an end-to-end case?
Yes. I have updated the FLIP. The beeline users can just use the connect
command to connect to the SQLGateway with the HiveServer2 endpoint.
For example, users just inputs "!connect
jdbc:hive2://<host>:<port>/<db>;auth=noSasl
hiveuser pass" into the terminal to connect to the SQLGateway.
> Is the streaming SQL supported? What's the behavior if I submit a
streaming query or I change the dialect to 'default'?
Yes. We don't limit the usage here. Users can switch to the streaming mode
or use the default dialect. But we don't suggest users use the hive
dialect in the streaming mode. As far as I know, it has some problems that
are not fixed yet, e.g. you may get errors for SQL that works in the batch
mode. I added a section to mention this.
> Considering the different users may have different requirements to
connect to different meta stores,
> they can use the DDL to register the HiveCatalog that satisfies their
requirements.
>> Could you give some examples to explain it more?
Hive supports setting multiple metastore addresses via the config option
"hive.metastore.urls". Here I just mean users can switch to connect to
different metastore instances using the CREATE CATALOG DDL. I updated the
FLIP to make it more clear.
Best,
Shengkai
godfrey he <go...@gmail.com> 于2022年6月6日周一 13:45写道：
> Hi Shengkai,
>
> Thanks for driving this.
>
> I have a few comments:
>
> Could you give a whole architecture about the Ecosystem of HiveServers
> and the SqlGateway, such as JDBC driver, Beeline, etc.
> Which is more clear for users.
>
> > Considering the different users may have different requirements to
> connect to different meta stores,
> > they can use the DDL to register the HiveCatalog that satisfies their
> requirements.
> Could you give some examples to explain it more?
>
> > How To Use
> Could you a complete example to describe an end-to-end case?
>
> Is the streaming sql supported? What's the behavior if I submit streaming
> query
> or I change the dialect to 'default'?
>
> Best,
> Godfrey
>
> Shengkai Fang <fs...@gmail.com> 于2022年6月1日周三 21:13写道：
> >
> > Hi, Jingsong.
> >
> > Thanks for your feedback.
> >
> > > I've read the FLIP and it's not quite clear what the specific
> unsupported
> > items are
> >
> > Yes. I have added a section named Difference with HiveServer2 and list
> the
> > difference between the SQL Gateway with HiveServer2 endpoint and
> > HiveServer2.
> >
> > > Support multiple metastore clients in one gateway?
> >
> > Yes. It may cause class conflicts when using the different versions of
> Hive
> > Catalog at the same time. I add a section named "How to use" to remind
> the
> > users don't use HiveCatalog with different versions together.
> >
> > > Hive versions and setup
> >
> > Considering the HiveServer2 endpoint binds to the HiveCatalog, we will
> not
> > introduce a new module about the HiveServer2 endpoint. The current
> > dependencies in the hive connector should be enough for the HiveServer2
> > Endpoint except for the hive-service-RPC(it contains the HiveServer2
> > interface). In this way, the hive connector jar will contain an
> endpoint. I
> > add a section named "Merge HiveServer2 Endpoint into Hive Connector
> > Module".
> >
> > For usage, the user can just add the hive connector jar into the
> classpath
> > and use the sql-gateway.sh to start the SQL Gateway with the hiveserver2
> > endpoint. You can refer to the section "How to use" for more details.
> >
> > Best,
> > Shengkai
> >
> > Jingsong Li <ji...@gmail.com> 于2022年6月1日周三 15:04写道：
> >
> > > Hi Shengkai,
> > >
> > > Thanks for driving.
> > >
> > > I have a few comments:
> > >
> > > ## Unsupported features
> > >
> > > I've read the FLIP and it's not quite clear what the specific
> unsupported
> > > items are?
> > > - For example, security related, is it not supported.
> > > - For example, is there a loss of precision for types
> > > - For example, the FetchResults are not the same
> > >
> > > ## Support multiple metastore clients in one gateway?
> > >
> > > > During the setup, the HiveServer2 tires to load the config in the
> > > hive-site.xml to initialize the Hive metastore client. In the Flink,
> we use
> > > the Catalog interface to connect to the Hive Metastore, which is
> allowed to
> > > communicate with different Hive Metastore[1]. Therefore, we allows the
> user
> > > to specify the path of the hive-site.xml as the endpoint parameters,
> which
> > > will used to create the default HiveCatalog in the Flink. Considering
> the
> > > different users may have different requirements to connect to different
> > > meta stores, they can use the DDL to register the HiveCatalog that
> > > satisfies their requirements.
> > >
> > > I understand it is difficult. You really want to support?
> > >
> > > ## Hive versions and setup
> > >
> > > I saw jark also commented, but FLIP does not seem to have been
> modified,
> > > how should the user setup, which jar to add, which hive metastore
> version
> > > to support? How to setup to support?
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fs...@gmail.com>
> wrote:
> > >
> > > > Hi, all.
> > > >
> > > > Considering we start to vote for FLIP-91 for a while, I think we can
> > > > restart the discussion about the FLIP-223.
> > > >
> > > > I am glad that you can give some feedback about FLIP-223.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > >
> > > > Martijn Visser <ma...@ververica.com> 于2022年5月6日周五 19:10写道：
> > > >
> > > > > Hi Shengkai,
> > > > >
> > > > > Thanks for clarifying.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com>
> wrote:
> > > > >
> > > > > > Hi Martijn.
> > > > > >
> > > > > > > So this implementation would not rely in any way on Hive, only
> on
> > > > > Thrift?
> > > > > >
> > > > > > Yes. The dependency is light. We also can just copy the iface
> file
> > > > from
> > > > > > the Hive repo and maintain by ourselves.
> > > > > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > > Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> > > > > >
> > > > > > > Hi Shengkai,
> > > > > > >
> > > > > > > > Actually we will only rely on the API in the Hive, which only
> > > > > contains
> > > > > > > the thrift file and the generated code
> > > > > > >
> > > > > > > So this implementation would not rely in any way on Hive, only
> on
> > > > > Thrift?
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn Visser
> > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > https://github.com/MartijnVisser
> > > > > > >
> > > > > > >
> > > > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fskmine@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi, Jark and Martijn
> > > > > > > >
> > > > > > > > Thanks for your feedback.
> > > > > > > >
> > > > > > > > > Kyuubi provides three ways to configure Hive metastore [1].
> > > Could
> > > > > we
> > > > > > > > provide similar abilities?
> > > > > > > >
> > > > > > > > Yes. I have updated the FLIP about this and it takes some
> time to
> > > > > > figure
> > > > > > > > out how the jdbc driver works. I added the section about how
> to
> > > use
> > > > > the
> > > > > > > > hive JDBC to configure the session-level catalog.
> > > > > > > >
> > > > > > > > > I think we can improve the "HiveServer2 Compatibility"
> section.
> > > > > > > >
> > > > > > > > Yes. I have updated the FLIP and added more details about the
> > > > > > > > compatibility.
> > > > > > > >
> > > > > > > > > Prefer to first complete the discussion and vote on
> FLIP-91
> > > then
> > > > > > > discuss
> > > > > > > > FLIP-223
> > > > > > > >
> > > > > > > > Of course. We can wait until the discussion of the FLIP-91
> > > > finishes.
> > > > > > > >
> > > > > > > > > Maintenance concerns about the hive
> > > > > > > >
> > > > > > > > Actually we will only rely on the API in the Hive, which only
> > > > > contains
> > > > > > > the
> > > > > > > > thrift file and the generated code[1]. I think it will not
> > > > influence
> > > > > us
> > > > > > > to
> > > > > > > > upgrade the java version.
> > > > > > > >
> > > > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shengkai
> > > > > > > >
> > > > > > > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二
> 20:44写道：
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'm not too familiar with Hive and HiveServer2, but I do
> have a
> > > > > > couple
> > > > > > > of
> > > > > > > > > questions/concerns:
> > > > > > > > >
> > > > > > > > > 1. What is the relationship between this FLIP and FLIP-91?
> My
> > > > > > > assumption
> > > > > > > > > would be that this FLIP (and therefore the HiveServer2)
> > > > > > implementation
> > > > > > > > > would need to be integrated in the REST Gateway, is that
> > > correct?
> > > > > If
> > > > > > > so,
> > > > > > > > I
> > > > > > > > > would prefer to first complete the discussion and vote on
> > > > FLIP-91,
> > > > > > else
> > > > > > > > > we'll have two moving FLIPs who have a direct relationship
> with
> > > > > each
> > > > > > > > other.
> > > > > > > > >
> > > > > > > > > 2. While I understand that Hive is important (in the
> Chinese
> > > > > > ecosystem,
> > > > > > > > not
> > > > > > > > > so much in Europe and the US), I still have maintenance
> > > concerns
> > > > on
> > > > > > > this
> > > > > > > > > topic. We know that the current Hive integration isn't
> exactly
> > > > > ideal
> > > > > > > and
> > > > > > > > > requires a lot of work to get in better shape. At the same
> > > time,
> > > > > Hive
> > > > > > > > still
> > > > > > > > > doesn't support Java 11 while we need (and should, given
> the
> > > > > premier
> > > > > > > > > support has ended already) to move away from Java 8.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > >
> > > > > > > > > Martijn Visser
> > > > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > > > https://github.com/MartijnVisser
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Thank Shengkai for driving this effort,
> > > > > > > > > > I think this is an essential addition to Flink Batch.
> > > > > > > > > >
> > > > > > > > > > I have some small suggestions:
> > > > > > > > > > 1) Kyuubi provides three ways to configure Hive metastore
> > > [1].
> > > > > > Could
> > > > > > > we
> > > > > > > > > > provide similar abilities?
> > > > > > > > > > Especially with the JDBC Connection URL, users can visit
> > > > > different
> > > > > > > Hive
> > > > > > > > > > metastore server instances.
> > > > > > > > > >
> > > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility"
> > > > > section.
> > > > > > > > > > We need to figure out two compatibility matrices. One is
> SQL
> > > > > > Gateway
> > > > > > > > with
> > > > > > > > > > different versions of Hive metastore,
> > > > > > > > > > and the other is different versions of Hive client (e.g.,
> > > Hive
> > > > > > JDBC)
> > > > > > > > with
> > > > > > > > > > SQL Gateway. We need to clarify
> > > > > > > > > > what metastore and client versions we support and how
> users
> > > > > > configure
> > > > > > > > the
> > > > > > > > > > versions.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Jark
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1]:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > > > > > > >
> > > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <
> > > fskmine@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Jiang.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your feedback!
> > > > > > > > > > >
> > > > > > > > > > > > Integrating the Hive ecosystem should not require
> > > changing
> > > > > the
> > > > > > > > > service
> > > > > > > > > > > interface
> > > > > > > > > > >
> > > > > > > > > > > I move the API change to the FLIP-91. But I think it's
> > > > possible
> > > > > > we
> > > > > > > > add
> > > > > > > > > > more
> > > > > > > > > > > interfaces to intergrate the new endpoints in the
> future
> > > > > because
> > > > > > > > every
> > > > > > > > > > > endpoints's functionality is different. For example,
> the
> > > REST
> > > > > > > > endpoint
> > > > > > > > > > > doen't support to fetch operation-level logs but the
> > > > > hiveserver2
> > > > > > > > > endpoint
> > > > > > > > > > > supports. In this case, we need to modify the shared
> > > > > > GatewayService
> > > > > > > > to
> > > > > > > > > > > support the functionality exposed by the new endpint.
> > > > > > > > > > >
> > > > > > > > > > > > How to support different Hive versions?
> > > > > > > > > > >
> > > > > > > > > > > Do you means to support the different HiveServer2
> version?
> > > > The
> > > > > > > > > > HiveServer2
> > > > > > > > > > > uses the version to guarantee the compatibility.
> During the
> > > > > > > > > openSession,
> > > > > > > > > > > the client and server will determine the protocol
> > > > > > > > > version(minimun(client
> > > > > > > > > > > version, hiveendpoint version)). After that the client
> and
> > > > the
> > > > > > > server
> > > > > > > > > > uses
> > > > > > > > > > > the determined version to communicate. In the
> HiveServer2
> > > > > > endpoint,
> > > > > > > > it
> > > > > > > > > > > determines how the endpoint deserialize the results
> and the
> > > > > > result
> > > > > > > > > > schema.
> > > > > > > > > > > I add a section about HiveServer2 compatiblity.
> > > > > > > > > > >
> > > > > > > > > > > > Could you please fully provide its definition
> including
> > > > input
> > > > > > > > > > parameters
> > > > > > > > > > > and the corresponding return value schema?
> > > > > > > > > > >
> > > > > > > > > > > Because we implements the interface exposed by the
> Hive.
> > > So I
> > > > > add
> > > > > > > the
> > > > > > > > > > file
> > > > > > > > > > > link to the HiveServer2 interfaces[1], which contains
> all
> > > > input
> > > > > > > > > > parameters
> > > > > > > > > > > and the results. Considering the file doesn't contain
> the
> > > > > output
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > Operation, I add the output schema for all the
> supported
> > > > > > Operation
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > FLIP, which is not covered in the link. Hope these can
> > > > address
> > > > > > your
> > > > > > > > > > > question.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shengkai
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Nicholas Jiang <ni...@apache.org>
> 于2022年4月22日周五
> > > > > 16:43写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi Shengkai.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for driving the proposal of HiveServer2
> Endpoint
> > > > > > support.
> > > > > > > > For
> > > > > > > > > > the
> > > > > > > > > > > > "GatewayService API Change", I don't think the
> motivation
> > > > for
> > > > > > > > > > supporting
> > > > > > > > > > > > HiveServer2 endpoint need to change the
> GatewayService
> > > API,
> > > > > in
> > > > > > > > other
> > > > > > > > > > > words,
> > > > > > > > > > > > integrating the Hive ecosystem should not require
> > > changing
> > > > > the
> > > > > > > > > service
> > > > > > > > > > > > interface. If you confirm to change GatewayService
> > > > interface,
> > > > > > > IMO,
> > > > > > > > > the
> > > > > > > > > > > > proposal could be discussed in FLIP-91 because the
> public
> > > > > > > > interfaces
> > > > > > > > > > are
> > > > > > > > > > > > defined in FLIP-91.
> > > > > > > > > > > >
> > > > > > > > > > > > In addtion, how to support different Hive versions
> and
> > > how
> > > > to
> > > > > > > > > guarantee
> > > > > > > > > > > > compatibility is not mentioned in the design. What's
> the
> > > > > > behavior
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > compatibility?
> > > > > > > > > > > >
> > > > > > > > > > > > Finally, for the public interfaces, could you please
> > > fully
> > > > > > > provide
> > > > > > > > > its
> > > > > > > > > > > > definition including input parameters and the
> > > corresponding
> > > > > > > return
> > > > > > > > > > value
> > > > > > > > > > > > schema?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Nicholas Jiang
> > > > > > > > > > > >
> > > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > > > > > > Hi, Flink developers.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I want to start a discussion about the FLIP-223:
> > > Support
> > > > > > > > > HiveServer2
> > > > > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift
> > > > > interface
> > > > > > > > > exposed
> > > > > > > > > > > by
> > > > > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools
> > > based
> > > > > on
> > > > > > > the
> > > > > > > > > > > > > HiveServer2 can also be seamlessly migrated to the
> > > Flink
> > > > > SQL
> > > > > > > > > Gateway.
> > > > > > > > > > > > After
> > > > > > > > > > > > > the FLIP finishes, the users can have almost the
> same
> > > > > > > experience
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as
> in
> > > the
> > > > > > > > > > HiveServer2.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I am glad that you can give some feedback about
> > > FLIP-223.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Shengkai
> > > > > > > > > > > > >
> > > > > > > > > > > > > [1]
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by godfrey he <go...@gmail.com>.
Hi Shengkai,
Thanks for driving this.
I have a few comments:
Could you give a whole architecture about the Ecosystem of HiveServers
and the SqlGateway, such as JDBC driver, Beeline, etc.
Which is more clear for users.
> Considering the different users may have different requirements to connect to different meta stores,
> they can use the DDL to register the HiveCatalog that satisfies their requirements.
Could you give some examples to explain it more?
> How To Use
Could you a complete example to describe an end-to-end case?
Is the streaming sql supported? What's the behavior if I submit streaming query
or I change the dialect to 'default'?
Best,
Godfrey
Shengkai Fang <fs...@gmail.com> 于2022年6月1日周三 21:13写道：
>
> Hi, Jingsong.
>
> Thanks for your feedback.
>
> > I've read the FLIP and it's not quite clear what the specific unsupported
> items are
>
> Yes. I have added a section named Difference with HiveServer2 and list the
> difference between the SQL Gateway with HiveServer2 endpoint and
> HiveServer2.
>
> > Support multiple metastore clients in one gateway?
>
> Yes. It may cause class conflicts when using the different versions of Hive
> Catalog at the same time. I add a section named "How to use" to remind the
> users don't use HiveCatalog with different versions together.
>
> > Hive versions and setup
>
> Considering the HiveServer2 endpoint binds to the HiveCatalog, we will not
> introduce a new module about the HiveServer2 endpoint. The current
> dependencies in the hive connector should be enough for the HiveServer2
> Endpoint except for the hive-service-RPC(it contains the HiveServer2
> interface). In this way, the hive connector jar will contain an endpoint. I
> add a section named "Merge HiveServer2 Endpoint into Hive Connector
> Module".
>
> For usage, the user can just add the hive connector jar into the classpath
> and use the sql-gateway.sh to start the SQL Gateway with the hiveserver2
> endpoint. You can refer to the section "How to use" for more details.
>
> Best,
> Shengkai
>
> Jingsong Li <ji...@gmail.com> 于2022年6月1日周三 15:04写道：
>
> > Hi Shengkai,
> >
> > Thanks for driving.
> >
> > I have a few comments:
> >
> > ## Unsupported features
> >
> > I've read the FLIP and it's not quite clear what the specific unsupported
> > items are?
> > - For example, security related, is it not supported.
> > - For example, is there a loss of precision for types
> > - For example, the FetchResults are not the same
> >
> > ## Support multiple metastore clients in one gateway?
> >
> > > During the setup, the HiveServer2 tires to load the config in the
> > hive-site.xml to initialize the Hive metastore client. In the Flink, we use
> > the Catalog interface to connect to the Hive Metastore, which is allowed to
> > communicate with different Hive Metastore[1]. Therefore, we allows the user
> > to specify the path of the hive-site.xml as the endpoint parameters, which
> > will used to create the default HiveCatalog in the Flink. Considering the
> > different users may have different requirements to connect to different
> > meta stores, they can use the DDL to register the HiveCatalog that
> > satisfies their requirements.
> >
> > I understand it is difficult. You really want to support?
> >
> > ## Hive versions and setup
> >
> > I saw jark also commented, but FLIP does not seem to have been modified,
> > how should the user setup, which jar to add, which hive metastore version
> > to support? How to setup to support?
> >
> > Best,
> > Jingsong
> >
> > On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fs...@gmail.com> wrote:
> >
> > > Hi, all.
> > >
> > > Considering we start to vote for FLIP-91 for a while, I think we can
> > > restart the discussion about the FLIP-223.
> > >
> > > I am glad that you can give some feedback about FLIP-223.
> > >
> > > Best,
> > > Shengkai
> > >
> > >
> > > Martijn Visser <ma...@ververica.com> 于2022年5月6日周五 19:10写道：
> > >
> > > > Hi Shengkai,
> > > >
> > > > Thanks for clarifying.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com> wrote:
> > > >
> > > > > Hi Martijn.
> > > > >
> > > > > > So this implementation would not rely in any way on Hive, only on
> > > > Thrift?
> > > > >
> > > > > Yes. The dependency is light. We also can just copy the iface file
> > > from
> > > > > the Hive repo and maintain by ourselves.
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> > > > >
> > > > > > Hi Shengkai,
> > > > > >
> > > > > > > Actually we will only rely on the API in the Hive, which only
> > > > contains
> > > > > > the thrift file and the generated code
> > > > > >
> > > > > > So this implementation would not rely in any way on Hive, only on
> > > > Thrift?
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn Visser
> > > > > > https://twitter.com/MartijnVisser82
> > > > > > https://github.com/MartijnVisser
> > > > > >
> > > > > >
> > > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi, Jark and Martijn
> > > > > > >
> > > > > > > Thanks for your feedback.
> > > > > > >
> > > > > > > > Kyuubi provides three ways to configure Hive metastore [1].
> > Could
> > > > we
> > > > > > > provide similar abilities?
> > > > > > >
> > > > > > > Yes. I have updated the FLIP about this and it takes some time to
> > > > > figure
> > > > > > > out how the jdbc driver works. I added the section about how to
> > use
> > > > the
> > > > > > > hive JDBC to configure the session-level catalog.
> > > > > > >
> > > > > > > > I think we can improve the "HiveServer2 Compatibility" section.
> > > > > > >
> > > > > > > Yes. I have updated the FLIP and added more details about the
> > > > > > > compatibility.
> > > > > > >
> > > > > > > > Prefer to first complete the discussion and vote on FLIP-91
> > then
> > > > > > discuss
> > > > > > > FLIP-223
> > > > > > >
> > > > > > > Of course. We can wait until the discussion of the FLIP-91
> > > finishes.
> > > > > > >
> > > > > > > > Maintenance concerns about the hive
> > > > > > >
> > > > > > > Actually we will only rely on the API in the Hive, which only
> > > > contains
> > > > > > the
> > > > > > > thrift file and the generated code[1]. I think it will not
> > > influence
> > > > us
> > > > > > to
> > > > > > > upgrade the java version.
> > > > > > >
> > > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
> > > > > > >
> > > > > > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I'm not too familiar with Hive and HiveServer2, but I do have a
> > > > > couple
> > > > > > of
> > > > > > > > questions/concerns:
> > > > > > > >
> > > > > > > > 1. What is the relationship between this FLIP and FLIP-91? My
> > > > > > assumption
> > > > > > > > would be that this FLIP (and therefore the HiveServer2)
> > > > > implementation
> > > > > > > > would need to be integrated in the REST Gateway, is that
> > correct?
> > > > If
> > > > > > so,
> > > > > > > I
> > > > > > > > would prefer to first complete the discussion and vote on
> > > FLIP-91,
> > > > > else
> > > > > > > > we'll have two moving FLIPs who have a direct relationship with
> > > > each
> > > > > > > other.
> > > > > > > >
> > > > > > > > 2. While I understand that Hive is important (in the Chinese
> > > > > ecosystem,
> > > > > > > not
> > > > > > > > so much in Europe and the US), I still have maintenance
> > concerns
> > > on
> > > > > > this
> > > > > > > > topic. We know that the current Hive integration isn't exactly
> > > > ideal
> > > > > > and
> > > > > > > > requires a lot of work to get in better shape. At the same
> > time,
> > > > Hive
> > > > > > > still
> > > > > > > > doesn't support Java 11 while we need (and should, given the
> > > > premier
> > > > > > > > support has ended already) to move away from Java 8.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > Martijn Visser
> > > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > > https://github.com/MartijnVisser
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com>
> > wrote:
> > > > > > > >
> > > > > > > > > Thank Shengkai for driving this effort,
> > > > > > > > > I think this is an essential addition to Flink Batch.
> > > > > > > > >
> > > > > > > > > I have some small suggestions:
> > > > > > > > > 1) Kyuubi provides three ways to configure Hive metastore
> > [1].
> > > > > Could
> > > > > > we
> > > > > > > > > provide similar abilities?
> > > > > > > > > Especially with the JDBC Connection URL, users can visit
> > > > different
> > > > > > Hive
> > > > > > > > > metastore server instances.
> > > > > > > > >
> > > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility"
> > > > section.
> > > > > > > > > We need to figure out two compatibility matrices. One is SQL
> > > > > Gateway
> > > > > > > with
> > > > > > > > > different versions of Hive metastore,
> > > > > > > > > and the other is different versions of Hive client (e.g.,
> > Hive
> > > > > JDBC)
> > > > > > > with
> > > > > > > > > SQL Gateway. We need to clarify
> > > > > > > > > what metastore and client versions we support and how users
> > > > > configure
> > > > > > > the
> > > > > > > > > versions.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Jark
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > > > > > >
> > > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <
> > fskmine@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi, Jiang.
> > > > > > > > > >
> > > > > > > > > > Thanks for your feedback!
> > > > > > > > > >
> > > > > > > > > > > Integrating the Hive ecosystem should not require
> > changing
> > > > the
> > > > > > > > service
> > > > > > > > > > interface
> > > > > > > > > >
> > > > > > > > > > I move the API change to the FLIP-91. But I think it's
> > > possible
> > > > > we
> > > > > > > add
> > > > > > > > > more
> > > > > > > > > > interfaces to intergrate the new endpoints in the future
> > > > because
> > > > > > > every
> > > > > > > > > > endpoints's functionality is different. For example, the
> > REST
> > > > > > > endpoint
> > > > > > > > > > doen't support to fetch operation-level logs but the
> > > > hiveserver2
> > > > > > > > endpoint
> > > > > > > > > > supports. In this case, we need to modify the shared
> > > > > GatewayService
> > > > > > > to
> > > > > > > > > > support the functionality exposed by the new endpint.
> > > > > > > > > >
> > > > > > > > > > > How to support different Hive versions?
> > > > > > > > > >
> > > > > > > > > > Do you means to support the different HiveServer2 version?
> > > The
> > > > > > > > > HiveServer2
> > > > > > > > > > uses the version to guarantee the compatibility. During the
> > > > > > > > openSession,
> > > > > > > > > > the client and server will determine the protocol
> > > > > > > > version(minimun(client
> > > > > > > > > > version, hiveendpoint version)). After that the client and
> > > the
> > > > > > server
> > > > > > > > > uses
> > > > > > > > > > the determined version to communicate. In the HiveServer2
> > > > > endpoint,
> > > > > > > it
> > > > > > > > > > determines how the endpoint deserialize the results and the
> > > > > result
> > > > > > > > > schema.
> > > > > > > > > > I add a section about HiveServer2 compatiblity.
> > > > > > > > > >
> > > > > > > > > > > Could you please fully provide its definition including
> > > input
> > > > > > > > > parameters
> > > > > > > > > > and the corresponding return value schema?
> > > > > > > > > >
> > > > > > > > > > Because we implements the interface exposed by the Hive.
> > So I
> > > > add
> > > > > > the
> > > > > > > > > file
> > > > > > > > > > link to the HiveServer2 interfaces[1], which contains all
> > > input
> > > > > > > > > parameters
> > > > > > > > > > and the results. Considering the file doesn't contain the
> > > > output
> > > > > > for
> > > > > > > > the
> > > > > > > > > > Operation, I add the output schema for all the supported
> > > > > Operation
> > > > > > in
> > > > > > > > the
> > > > > > > > > > FLIP, which is not covered in the link. Hope these can
> > > address
> > > > > your
> > > > > > > > > > question.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shengkai
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五
> > > > 16:43写道：
> > > > > > > > > >
> > > > > > > > > > > Hi Shengkai.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint
> > > > > support.
> > > > > > > For
> > > > > > > > > the
> > > > > > > > > > > "GatewayService API Change", I don't think the motivation
> > > for
> > > > > > > > > supporting
> > > > > > > > > > > HiveServer2 endpoint need to change the GatewayService
> > API,
> > > > in
> > > > > > > other
> > > > > > > > > > words,
> > > > > > > > > > > integrating the Hive ecosystem should not require
> > changing
> > > > the
> > > > > > > > service
> > > > > > > > > > > interface. If you confirm to change GatewayService
> > > interface,
> > > > > > IMO,
> > > > > > > > the
> > > > > > > > > > > proposal could be discussed in FLIP-91 because the public
> > > > > > > interfaces
> > > > > > > > > are
> > > > > > > > > > > defined in FLIP-91.
> > > > > > > > > > >
> > > > > > > > > > > In addtion, how to support different Hive versions and
> > how
> > > to
> > > > > > > > guarantee
> > > > > > > > > > > compatibility is not mentioned in the design. What's the
> > > > > behavior
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > compatibility?
> > > > > > > > > > >
> > > > > > > > > > > Finally, for the public interfaces, could you please
> > fully
> > > > > > provide
> > > > > > > > its
> > > > > > > > > > > definition including input parameters and the
> > corresponding
> > > > > > return
> > > > > > > > > value
> > > > > > > > > > > schema?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Nicholas Jiang
> > > > > > > > > > >
> > > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > > > > > Hi, Flink developers.
> > > > > > > > > > > >
> > > > > > > > > > > > I want to start a discussion about the FLIP-223:
> > Support
> > > > > > > > HiveServer2
> > > > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift
> > > > interface
> > > > > > > > exposed
> > > > > > > > > > by
> > > > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools
> > based
> > > > on
> > > > > > the
> > > > > > > > > > > > HiveServer2 can also be seamlessly migrated to the
> > Flink
> > > > SQL
> > > > > > > > Gateway.
> > > > > > > > > > > After
> > > > > > > > > > > > the FLIP finishes, the users can have almost the same
> > > > > > experience
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in
> > the
> > > > > > > > > HiveServer2.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I am glad that you can give some feedback about
> > FLIP-223.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Shengkai
> > > > > > > > > > > >
> > > > > > > > > > > > [1]
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Shengkai Fang <fs...@gmail.com>.
Hi, Jingsong.
Thanks for your feedback.
> I've read the FLIP and it's not quite clear what the specific unsupported
items are
Yes. I have added a section named Difference with HiveServer2 and list the
difference between the SQL Gateway with HiveServer2 endpoint and
HiveServer2.
> Support multiple metastore clients in one gateway?
Yes. It may cause class conflicts when using the different versions of Hive
Catalog at the same time. I add a section named "How to use" to remind the
users don't use HiveCatalog with different versions together.
> Hive versions and setup
Considering the HiveServer2 endpoint binds to the HiveCatalog, we will not
introduce a new module about the HiveServer2 endpoint. The current
dependencies in the hive connector should be enough for the HiveServer2
Endpoint except for the hive-service-RPC(it contains the HiveServer2
interface). In this way, the hive connector jar will contain an endpoint. I
add a section named "Merge HiveServer2 Endpoint into Hive Connector
Module".
For usage, the user can just add the hive connector jar into the classpath
and use the sql-gateway.sh to start the SQL Gateway with the hiveserver2
endpoint. You can refer to the section "How to use" for more details.
Best,
Shengkai
Jingsong Li <ji...@gmail.com> 于2022年6月1日周三 15:04写道：
> Hi Shengkai,
>
> Thanks for driving.
>
> I have a few comments:
>
> ## Unsupported features
>
> I've read the FLIP and it's not quite clear what the specific unsupported
> items are?
> - For example, security related, is it not supported.
> - For example, is there a loss of precision for types
> - For example, the FetchResults are not the same
>
> ## Support multiple metastore clients in one gateway?
>
> > During the setup, the HiveServer2 tires to load the config in the
> hive-site.xml to initialize the Hive metastore client. In the Flink, we use
> the Catalog interface to connect to the Hive Metastore, which is allowed to
> communicate with different Hive Metastore[1]. Therefore, we allows the user
> to specify the path of the hive-site.xml as the endpoint parameters, which
> will used to create the default HiveCatalog in the Flink. Considering the
> different users may have different requirements to connect to different
> meta stores, they can use the DDL to register the HiveCatalog that
> satisfies their requirements.
>
> I understand it is difficult. You really want to support?
>
> ## Hive versions and setup
>
> I saw jark also commented, but FLIP does not seem to have been modified,
> how should the user setup, which jar to add, which hive metastore version
> to support? How to setup to support?
>
> Best,
> Jingsong
>
> On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fs...@gmail.com> wrote:
>
> > Hi, all.
> >
> > Considering we start to vote for FLIP-91 for a while, I think we can
> > restart the discussion about the FLIP-223.
> >
> > I am glad that you can give some feedback about FLIP-223.
> >
> > Best,
> > Shengkai
> >
> >
> > Martijn Visser <ma...@ververica.com> 于2022年5月6日周五 19:10写道：
> >
> > > Hi Shengkai,
> > >
> > > Thanks for clarifying.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com> wrote:
> > >
> > > > Hi Martijn.
> > > >
> > > > > So this implementation would not rely in any way on Hive, only on
> > > Thrift?
> > > >
> > > > Yes. The dependency is light. We also can just copy the iface file
> > from
> > > > the Hive repo and maintain by ourselves.
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> > > >
> > > > > Hi Shengkai,
> > > > >
> > > > > > Actually we will only rely on the API in the Hive, which only
> > > contains
> > > > > the thrift file and the generated code
> > > > >
> > > > > So this implementation would not rely in any way on Hive, only on
> > > Thrift?
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn Visser
> > > > > https://twitter.com/MartijnVisser82
> > > > > https://github.com/MartijnVisser
> > > > >
> > > > >
> > > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi, Jark and Martijn
> > > > > >
> > > > > > Thanks for your feedback.
> > > > > >
> > > > > > > Kyuubi provides three ways to configure Hive metastore [1].
> Could
> > > we
> > > > > > provide similar abilities?
> > > > > >
> > > > > > Yes. I have updated the FLIP about this and it takes some time to
> > > > figure
> > > > > > out how the jdbc driver works. I added the section about how to
> use
> > > the
> > > > > > hive JDBC to configure the session-level catalog.
> > > > > >
> > > > > > > I think we can improve the "HiveServer2 Compatibility" section.
> > > > > >
> > > > > > Yes. I have updated the FLIP and added more details about the
> > > > > > compatibility.
> > > > > >
> > > > > > > Prefer to first complete the discussion and vote on FLIP-91
> then
> > > > > discuss
> > > > > > FLIP-223
> > > > > >
> > > > > > Of course. We can wait until the discussion of the FLIP-91
> > finishes.
> > > > > >
> > > > > > > Maintenance concerns about the hive
> > > > > >
> > > > > > Actually we will only rely on the API in the Hive, which only
> > > contains
> > > > > the
> > > > > > thrift file and the generated code[1]. I think it will not
> > influence
> > > us
> > > > > to
> > > > > > upgrade the java version.
> > > > > >
> > > > > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > > > > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm not too familiar with Hive and HiveServer2, but I do have a
> > > > couple
> > > > > of
> > > > > > > questions/concerns:
> > > > > > >
> > > > > > > 1. What is the relationship between this FLIP and FLIP-91? My
> > > > > assumption
> > > > > > > would be that this FLIP (and therefore the HiveServer2)
> > > > implementation
> > > > > > > would need to be integrated in the REST Gateway, is that
> correct?
> > > If
> > > > > so,
> > > > > > I
> > > > > > > would prefer to first complete the discussion and vote on
> > FLIP-91,
> > > > else
> > > > > > > we'll have two moving FLIPs who have a direct relationship with
> > > each
> > > > > > other.
> > > > > > >
> > > > > > > 2. While I understand that Hive is important (in the Chinese
> > > > ecosystem,
> > > > > > not
> > > > > > > so much in Europe and the US), I still have maintenance
> concerns
> > on
> > > > > this
> > > > > > > topic. We know that the current Hive integration isn't exactly
> > > ideal
> > > > > and
> > > > > > > requires a lot of work to get in better shape. At the same
> time,
> > > Hive
> > > > > > still
> > > > > > > doesn't support Java 11 while we need (and should, given the
> > > premier
> > > > > > > support has ended already) to move away from Java 8.
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > > Martijn Visser
> > > > > > > https://twitter.com/MartijnVisser82
> > > > > > > https://github.com/MartijnVisser
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com>
> wrote:
> > > > > > >
> > > > > > > > Thank Shengkai for driving this effort,
> > > > > > > > I think this is an essential addition to Flink Batch.
> > > > > > > >
> > > > > > > > I have some small suggestions:
> > > > > > > > 1) Kyuubi provides three ways to configure Hive metastore
> [1].
> > > > Could
> > > > > we
> > > > > > > > provide similar abilities?
> > > > > > > > Especially with the JDBC Connection URL, users can visit
> > > different
> > > > > Hive
> > > > > > > > metastore server instances.
> > > > > > > >
> > > > > > > > 2) I think we can improve the "HiveServer2 Compatibility"
> > > section.
> > > > > > > > We need to figure out two compatibility matrices. One is SQL
> > > > Gateway
> > > > > > with
> > > > > > > > different versions of Hive metastore,
> > > > > > > > and the other is different versions of Hive client (e.g.,
> Hive
> > > > JDBC)
> > > > > > with
> > > > > > > > SQL Gateway. We need to clarify
> > > > > > > > what metastore and client versions we support and how users
> > > > configure
> > > > > > the
> > > > > > > > versions.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jark
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > > > > >
> > > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <
> fskmine@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi, Jiang.
> > > > > > > > >
> > > > > > > > > Thanks for your feedback!
> > > > > > > > >
> > > > > > > > > > Integrating the Hive ecosystem should not require
> changing
> > > the
> > > > > > > service
> > > > > > > > > interface
> > > > > > > > >
> > > > > > > > > I move the API change to the FLIP-91. But I think it's
> > possible
> > > > we
> > > > > > add
> > > > > > > > more
> > > > > > > > > interfaces to intergrate the new endpoints in the future
> > > because
> > > > > > every
> > > > > > > > > endpoints's functionality is different. For example, the
> REST
> > > > > > endpoint
> > > > > > > > > doen't support to fetch operation-level logs but the
> > > hiveserver2
> > > > > > > endpoint
> > > > > > > > > supports. In this case, we need to modify the shared
> > > > GatewayService
> > > > > > to
> > > > > > > > > support the functionality exposed by the new endpint.
> > > > > > > > >
> > > > > > > > > > How to support different Hive versions?
> > > > > > > > >
> > > > > > > > > Do you means to support the different HiveServer2 version?
> > The
> > > > > > > > HiveServer2
> > > > > > > > > uses the version to guarantee the compatibility. During the
> > > > > > > openSession,
> > > > > > > > > the client and server will determine the protocol
> > > > > > > version(minimun(client
> > > > > > > > > version, hiveendpoint version)). After that the client and
> > the
> > > > > server
> > > > > > > > uses
> > > > > > > > > the determined version to communicate. In the HiveServer2
> > > > endpoint,
> > > > > > it
> > > > > > > > > determines how the endpoint deserialize the results and the
> > > > result
> > > > > > > > schema.
> > > > > > > > > I add a section about HiveServer2 compatiblity.
> > > > > > > > >
> > > > > > > > > > Could you please fully provide its definition including
> > input
> > > > > > > > parameters
> > > > > > > > > and the corresponding return value schema?
> > > > > > > > >
> > > > > > > > > Because we implements the interface exposed by the Hive.
> So I
> > > add
> > > > > the
> > > > > > > > file
> > > > > > > > > link to the HiveServer2 interfaces[1], which contains all
> > input
> > > > > > > > parameters
> > > > > > > > > and the results. Considering the file doesn't contain the
> > > output
> > > > > for
> > > > > > > the
> > > > > > > > > Operation, I add the output schema for all the supported
> > > > Operation
> > > > > in
> > > > > > > the
> > > > > > > > > FLIP, which is not covered in the link. Hope these can
> > address
> > > > your
> > > > > > > > > question.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shengkai
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五
> > > 16:43写道：
> > > > > > > > >
> > > > > > > > > > Hi Shengkai.
> > > > > > > > > >
> > > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint
> > > > support.
> > > > > > For
> > > > > > > > the
> > > > > > > > > > "GatewayService API Change", I don't think the motivation
> > for
> > > > > > > > supporting
> > > > > > > > > > HiveServer2 endpoint need to change the GatewayService
> API,
> > > in
> > > > > > other
> > > > > > > > > words,
> > > > > > > > > > integrating the Hive ecosystem should not require
> changing
> > > the
> > > > > > > service
> > > > > > > > > > interface. If you confirm to change GatewayService
> > interface,
> > > > > IMO,
> > > > > > > the
> > > > > > > > > > proposal could be discussed in FLIP-91 because the public
> > > > > > interfaces
> > > > > > > > are
> > > > > > > > > > defined in FLIP-91.
> > > > > > > > > >
> > > > > > > > > > In addtion, how to support different Hive versions and
> how
> > to
> > > > > > > guarantee
> > > > > > > > > > compatibility is not mentioned in the design. What's the
> > > > behavior
> > > > > > of
> > > > > > > > the
> > > > > > > > > > compatibility?
> > > > > > > > > >
> > > > > > > > > > Finally, for the public interfaces, could you please
> fully
> > > > > provide
> > > > > > > its
> > > > > > > > > > definition including input parameters and the
> corresponding
> > > > > return
> > > > > > > > value
> > > > > > > > > > schema?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Nicholas Jiang
> > > > > > > > > >
> > > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > > > > Hi, Flink developers.
> > > > > > > > > > >
> > > > > > > > > > > I want to start a discussion about the FLIP-223:
> Support
> > > > > > > HiveServer2
> > > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift
> > > interface
> > > > > > > exposed
> > > > > > > > > by
> > > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools
> based
> > > on
> > > > > the
> > > > > > > > > > > HiveServer2 can also be seamlessly migrated to the
> Flink
> > > SQL
> > > > > > > Gateway.
> > > > > > > > > > After
> > > > > > > > > > > the FLIP finishes, the users can have almost the same
> > > > > experience
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in
> the
> > > > > > > > HiveServer2.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I am glad that you can give some feedback about
> FLIP-223.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Shengkai
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Jingsong Li <ji...@gmail.com>.
Hi Shengkai,
Thanks for driving.
I have a few comments:
## Unsupported features
I've read the FLIP and it's not quite clear what the specific unsupported
items are?
- For example, security related, is it not supported.
- For example, is there a loss of precision for types
- For example, the FetchResults are not the same
## Support multiple metastore clients in one gateway?
> During the setup, the HiveServer2 tires to load the config in the
hive-site.xml to initialize the Hive metastore client. In the Flink, we use
the Catalog interface to connect to the Hive Metastore, which is allowed to
communicate with different Hive Metastore[1]. Therefore, we allows the user
to specify the path of the hive-site.xml as the endpoint parameters, which
will used to create the default HiveCatalog in the Flink. Considering the
different users may have different requirements to connect to different
meta stores, they can use the DDL to register the HiveCatalog that
satisfies their requirements.
I understand it is difficult. You really want to support?
## Hive versions and setup
I saw jark also commented, but FLIP does not seem to have been modified,
how should the user setup, which jar to add, which hive metastore version
to support? How to setup to support?
Best,
Jingsong
On Tue, May 24, 2022 at 11:57 AM Shengkai Fang <fs...@gmail.com> wrote:
> Hi, all.
>
> Considering we start to vote for FLIP-91 for a while, I think we can
> restart the discussion about the FLIP-223.
>
> I am glad that you can give some feedback about FLIP-223.
>
> Best,
> Shengkai
>
>
> Martijn Visser <ma...@ververica.com> 于2022年5月6日周五 19:10写道：
>
> > Hi Shengkai,
> >
> > Thanks for clarifying.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com> wrote:
> >
> > > Hi Martijn.
> > >
> > > > So this implementation would not rely in any way on Hive, only on
> > Thrift?
> > >
> > > Yes. The dependency is light. We also can just copy the iface file
> from
> > > the Hive repo and maintain by ourselves.
> > >
> > > Best,
> > > Shengkai
> > >
> > > Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> > >
> > > > Hi Shengkai,
> > > >
> > > > > Actually we will only rely on the API in the Hive, which only
> > contains
> > > > the thrift file and the generated code
> > > >
> > > > So this implementation would not rely in any way on Hive, only on
> > Thrift?
> > > >
> > > > Best regards,
> > > >
> > > > Martijn Visser
> > > > https://twitter.com/MartijnVisser82
> > > > https://github.com/MartijnVisser
> > > >
> > > >
> > > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com>
> wrote:
> > > >
> > > > > Hi, Jark and Martijn
> > > > >
> > > > > Thanks for your feedback.
> > > > >
> > > > > > Kyuubi provides three ways to configure Hive metastore [1]. Could
> > we
> > > > > provide similar abilities?
> > > > >
> > > > > Yes. I have updated the FLIP about this and it takes some time to
> > > figure
> > > > > out how the jdbc driver works. I added the section about how to use
> > the
> > > > > hive JDBC to configure the session-level catalog.
> > > > >
> > > > > > I think we can improve the "HiveServer2 Compatibility" section.
> > > > >
> > > > > Yes. I have updated the FLIP and added more details about the
> > > > > compatibility.
> > > > >
> > > > > > Prefer to first complete the discussion and vote on FLIP-91 then
> > > > discuss
> > > > > FLIP-223
> > > > >
> > > > > Of course. We can wait until the discussion of the FLIP-91
> finishes.
> > > > >
> > > > > > Maintenance concerns about the hive
> > > > >
> > > > > Actually we will only rely on the API in the Hive, which only
> > contains
> > > > the
> > > > > thrift file and the generated code[1]. I think it will not
> influence
> > us
> > > > to
> > > > > upgrade the java version.
> > > > >
> > > > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm not too familiar with Hive and HiveServer2, but I do have a
> > > couple
> > > > of
> > > > > > questions/concerns:
> > > > > >
> > > > > > 1. What is the relationship between this FLIP and FLIP-91? My
> > > > assumption
> > > > > > would be that this FLIP (and therefore the HiveServer2)
> > > implementation
> > > > > > would need to be integrated in the REST Gateway, is that correct?
> > If
> > > > so,
> > > > > I
> > > > > > would prefer to first complete the discussion and vote on
> FLIP-91,
> > > else
> > > > > > we'll have two moving FLIPs who have a direct relationship with
> > each
> > > > > other.
> > > > > >
> > > > > > 2. While I understand that Hive is important (in the Chinese
> > > ecosystem,
> > > > > not
> > > > > > so much in Europe and the US), I still have maintenance concerns
> on
> > > > this
> > > > > > topic. We know that the current Hive integration isn't exactly
> > ideal
> > > > and
> > > > > > requires a lot of work to get in better shape. At the same time,
> > Hive
> > > > > still
> > > > > > doesn't support Java 11 while we need (and should, given the
> > premier
> > > > > > support has ended already) to move away from Java 8.
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn Visser
> > > > > > https://twitter.com/MartijnVisser82
> > > > > > https://github.com/MartijnVisser
> > > > > >
> > > > > >
> > > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com> wrote:
> > > > > >
> > > > > > > Thank Shengkai for driving this effort,
> > > > > > > I think this is an essential addition to Flink Batch.
> > > > > > >
> > > > > > > I have some small suggestions:
> > > > > > > 1) Kyuubi provides three ways to configure Hive metastore [1].
> > > Could
> > > > we
> > > > > > > provide similar abilities?
> > > > > > > Especially with the JDBC Connection URL, users can visit
> > different
> > > > Hive
> > > > > > > metastore server instances.
> > > > > > >
> > > > > > > 2) I think we can improve the "HiveServer2 Compatibility"
> > section.
> > > > > > > We need to figure out two compatibility matrices. One is SQL
> > > Gateway
> > > > > with
> > > > > > > different versions of Hive metastore,
> > > > > > > and the other is different versions of Hive client (e.g., Hive
> > > JDBC)
> > > > > with
> > > > > > > SQL Gateway. We need to clarify
> > > > > > > what metastore and client versions we support and how users
> > > configure
> > > > > the
> > > > > > > versions.
> > > > > > >
> > > > > > > Best,
> > > > > > > Jark
> > > > > > >
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > > > >
> > > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fskmine@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi, Jiang.
> > > > > > > >
> > > > > > > > Thanks for your feedback!
> > > > > > > >
> > > > > > > > > Integrating the Hive ecosystem should not require changing
> > the
> > > > > > service
> > > > > > > > interface
> > > > > > > >
> > > > > > > > I move the API change to the FLIP-91. But I think it's
> possible
> > > we
> > > > > add
> > > > > > > more
> > > > > > > > interfaces to intergrate the new endpoints in the future
> > because
> > > > > every
> > > > > > > > endpoints's functionality is different. For example, the REST
> > > > > endpoint
> > > > > > > > doen't support to fetch operation-level logs but the
> > hiveserver2
> > > > > > endpoint
> > > > > > > > supports. In this case, we need to modify the shared
> > > GatewayService
> > > > > to
> > > > > > > > support the functionality exposed by the new endpint.
> > > > > > > >
> > > > > > > > > How to support different Hive versions?
> > > > > > > >
> > > > > > > > Do you means to support the different HiveServer2 version?
> The
> > > > > > > HiveServer2
> > > > > > > > uses the version to guarantee the compatibility. During the
> > > > > > openSession,
> > > > > > > > the client and server will determine the protocol
> > > > > > version(minimun(client
> > > > > > > > version, hiveendpoint version)). After that the client and
> the
> > > > server
> > > > > > > uses
> > > > > > > > the determined version to communicate. In the HiveServer2
> > > endpoint,
> > > > > it
> > > > > > > > determines how the endpoint deserialize the results and the
> > > result
> > > > > > > schema.
> > > > > > > > I add a section about HiveServer2 compatiblity.
> > > > > > > >
> > > > > > > > > Could you please fully provide its definition including
> input
> > > > > > > parameters
> > > > > > > > and the corresponding return value schema?
> > > > > > > >
> > > > > > > > Because we implements the interface exposed by the Hive. So I
> > add
> > > > the
> > > > > > > file
> > > > > > > > link to the HiveServer2 interfaces[1], which contains all
> input
> > > > > > > parameters
> > > > > > > > and the results. Considering the file doesn't contain the
> > output
> > > > for
> > > > > > the
> > > > > > > > Operation, I add the output schema for all the supported
> > > Operation
> > > > in
> > > > > > the
> > > > > > > > FLIP, which is not covered in the link. Hope these can
> address
> > > your
> > > > > > > > question.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shengkai
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五
> > 16:43写道：
> > > > > > > >
> > > > > > > > > Hi Shengkai.
> > > > > > > > >
> > > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint
> > > support.
> > > > > For
> > > > > > > the
> > > > > > > > > "GatewayService API Change", I don't think the motivation
> for
> > > > > > > supporting
> > > > > > > > > HiveServer2 endpoint need to change the GatewayService API,
> > in
> > > > > other
> > > > > > > > words,
> > > > > > > > > integrating the Hive ecosystem should not require changing
> > the
> > > > > > service
> > > > > > > > > interface. If you confirm to change GatewayService
> interface,
> > > > IMO,
> > > > > > the
> > > > > > > > > proposal could be discussed in FLIP-91 because the public
> > > > > interfaces
> > > > > > > are
> > > > > > > > > defined in FLIP-91.
> > > > > > > > >
> > > > > > > > > In addtion, how to support different Hive versions and how
> to
> > > > > > guarantee
> > > > > > > > > compatibility is not mentioned in the design. What's the
> > > behavior
> > > > > of
> > > > > > > the
> > > > > > > > > compatibility?
> > > > > > > > >
> > > > > > > > > Finally, for the public interfaces, could you please fully
> > > > provide
> > > > > > its
> > > > > > > > > definition including input parameters and the corresponding
> > > > return
> > > > > > > value
> > > > > > > > > schema?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Nicholas Jiang
> > > > > > > > >
> > > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > > > Hi, Flink developers.
> > > > > > > > > >
> > > > > > > > > > I want to start a discussion about the FLIP-223: Support
> > > > > > HiveServer2
> > > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift
> > interface
> > > > > > exposed
> > > > > > > > by
> > > > > > > > > > the HiveServer2, and users' BI, CLI and other tools based
> > on
> > > > the
> > > > > > > > > > HiveServer2 can also be seamlessly migrated to the Flink
> > SQL
> > > > > > Gateway.
> > > > > > > > > After
> > > > > > > > > > the FLIP finishes, the users can have almost the same
> > > > experience
> > > > > in
> > > > > > > the
> > > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the
> > > > > > > HiveServer2.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I am glad that you can give some feedback about FLIP-223.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Shengkai
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Shengkai Fang <fs...@gmail.com>.
Hi, all.
Considering we start to vote for FLIP-91 for a while, I think we can
restart the discussion about the FLIP-223.
I am glad that you can give some feedback about FLIP-223.
Best,
Shengkai
Martijn Visser <ma...@ververica.com> 于2022年5月6日周五 19:10写道：
> Hi Shengkai,
>
> Thanks for clarifying.
>
> Best regards,
>
> Martijn
>
> On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com> wrote:
>
> > Hi Martijn.
> >
> > > So this implementation would not rely in any way on Hive, only on
> Thrift?
> >
> > Yes. The dependency is light. We also can just copy the iface file from
> > the Hive repo and maintain by ourselves.
> >
> > Best,
> > Shengkai
> >
> > Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> >
> > > Hi Shengkai,
> > >
> > > > Actually we will only rely on the API in the Hive, which only
> contains
> > > the thrift file and the generated code
> > >
> > > So this implementation would not rely in any way on Hive, only on
> Thrift?
> > >
> > > Best regards,
> > >
> > > Martijn Visser
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> > >
> > >
> > > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com> wrote:
> > >
> > > > Hi, Jark and Martijn
> > > >
> > > > Thanks for your feedback.
> > > >
> > > > > Kyuubi provides three ways to configure Hive metastore [1]. Could
> we
> > > > provide similar abilities?
> > > >
> > > > Yes. I have updated the FLIP about this and it takes some time to
> > figure
> > > > out how the jdbc driver works. I added the section about how to use
> the
> > > > hive JDBC to configure the session-level catalog.
> > > >
> > > > > I think we can improve the "HiveServer2 Compatibility" section.
> > > >
> > > > Yes. I have updated the FLIP and added more details about the
> > > > compatibility.
> > > >
> > > > > Prefer to first complete the discussion and vote on FLIP-91 then
> > > discuss
> > > > FLIP-223
> > > >
> > > > Of course. We can wait until the discussion of the FLIP-91 finishes.
> > > >
> > > > > Maintenance concerns about the hive
> > > >
> > > > Actually we will only rely on the API in the Hive, which only
> contains
> > > the
> > > > thrift file and the generated code[1]. I think it will not influence
> us
> > > to
> > > > upgrade the java version.
> > > >
> > > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > > >
> > > > Best,
> > > > Shengkai
> > > >
> > > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm not too familiar with Hive and HiveServer2, but I do have a
> > couple
> > > of
> > > > > questions/concerns:
> > > > >
> > > > > 1. What is the relationship between this FLIP and FLIP-91? My
> > > assumption
> > > > > would be that this FLIP (and therefore the HiveServer2)
> > implementation
> > > > > would need to be integrated in the REST Gateway, is that correct?
> If
> > > so,
> > > > I
> > > > > would prefer to first complete the discussion and vote on FLIP-91,
> > else
> > > > > we'll have two moving FLIPs who have a direct relationship with
> each
> > > > other.
> > > > >
> > > > > 2. While I understand that Hive is important (in the Chinese
> > ecosystem,
> > > > not
> > > > > so much in Europe and the US), I still have maintenance concerns on
> > > this
> > > > > topic. We know that the current Hive integration isn't exactly
> ideal
> > > and
> > > > > requires a lot of work to get in better shape. At the same time,
> Hive
> > > > still
> > > > > doesn't support Java 11 while we need (and should, given the
> premier
> > > > > support has ended already) to move away from Java 8.
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn Visser
> > > > > https://twitter.com/MartijnVisser82
> > > > > https://github.com/MartijnVisser
> > > > >
> > > > >
> > > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com> wrote:
> > > > >
> > > > > > Thank Shengkai for driving this effort,
> > > > > > I think this is an essential addition to Flink Batch.
> > > > > >
> > > > > > I have some small suggestions:
> > > > > > 1) Kyuubi provides three ways to configure Hive metastore [1].
> > Could
> > > we
> > > > > > provide similar abilities?
> > > > > > Especially with the JDBC Connection URL, users can visit
> different
> > > Hive
> > > > > > metastore server instances.
> > > > > >
> > > > > > 2) I think we can improve the "HiveServer2 Compatibility"
> section.
> > > > > > We need to figure out two compatibility matrices. One is SQL
> > Gateway
> > > > with
> > > > > > different versions of Hive metastore,
> > > > > > and the other is different versions of Hive client (e.g., Hive
> > JDBC)
> > > > with
> > > > > > SQL Gateway. We need to clarify
> > > > > > what metastore and client versions we support and how users
> > configure
> > > > the
> > > > > > versions.
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > > >
> > > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fs...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi, Jiang.
> > > > > > >
> > > > > > > Thanks for your feedback!
> > > > > > >
> > > > > > > > Integrating the Hive ecosystem should not require changing
> the
> > > > > service
> > > > > > > interface
> > > > > > >
> > > > > > > I move the API change to the FLIP-91. But I think it's possible
> > we
> > > > add
> > > > > > more
> > > > > > > interfaces to intergrate the new endpoints in the future
> because
> > > > every
> > > > > > > endpoints's functionality is different. For example, the REST
> > > > endpoint
> > > > > > > doen't support to fetch operation-level logs but the
> hiveserver2
> > > > > endpoint
> > > > > > > supports. In this case, we need to modify the shared
> > GatewayService
> > > > to
> > > > > > > support the functionality exposed by the new endpint.
> > > > > > >
> > > > > > > > How to support different Hive versions?
> > > > > > >
> > > > > > > Do you means to support the different HiveServer2 version? The
> > > > > > HiveServer2
> > > > > > > uses the version to guarantee the compatibility. During the
> > > > > openSession,
> > > > > > > the client and server will determine the protocol
> > > > > version(minimun(client
> > > > > > > version, hiveendpoint version)). After that the client and the
> > > server
> > > > > > uses
> > > > > > > the determined version to communicate. In the HiveServer2
> > endpoint,
> > > > it
> > > > > > > determines how the endpoint deserialize the results and the
> > result
> > > > > > schema.
> > > > > > > I add a section about HiveServer2 compatiblity.
> > > > > > >
> > > > > > > > Could you please fully provide its definition including input
> > > > > > parameters
> > > > > > > and the corresponding return value schema?
> > > > > > >
> > > > > > > Because we implements the interface exposed by the Hive. So I
> add
> > > the
> > > > > > file
> > > > > > > link to the HiveServer2 interfaces[1], which contains all input
> > > > > > parameters
> > > > > > > and the results. Considering the file doesn't contain the
> output
> > > for
> > > > > the
> > > > > > > Operation, I add the output schema for all the supported
> > Operation
> > > in
> > > > > the
> > > > > > > FLIP, which is not covered in the link. Hope these can address
> > your
> > > > > > > question.
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五
> 16:43写道：
> > > > > > >
> > > > > > > > Hi Shengkai.
> > > > > > > >
> > > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint
> > support.
> > > > For
> > > > > > the
> > > > > > > > "GatewayService API Change", I don't think the motivation for
> > > > > > supporting
> > > > > > > > HiveServer2 endpoint need to change the GatewayService API,
> in
> > > > other
> > > > > > > words,
> > > > > > > > integrating the Hive ecosystem should not require changing
> the
> > > > > service
> > > > > > > > interface. If you confirm to change GatewayService interface,
> > > IMO,
> > > > > the
> > > > > > > > proposal could be discussed in FLIP-91 because the public
> > > > interfaces
> > > > > > are
> > > > > > > > defined in FLIP-91.
> > > > > > > >
> > > > > > > > In addtion, how to support different Hive versions and how to
> > > > > guarantee
> > > > > > > > compatibility is not mentioned in the design. What's the
> > behavior
> > > > of
> > > > > > the
> > > > > > > > compatibility?
> > > > > > > >
> > > > > > > > Finally, for the public interfaces, could you please fully
> > > provide
> > > > > its
> > > > > > > > definition including input parameters and the corresponding
> > > return
> > > > > > value
> > > > > > > > schema?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Nicholas Jiang
> > > > > > > >
> > > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > > Hi, Flink developers.
> > > > > > > > >
> > > > > > > > > I want to start a discussion about the FLIP-223: Support
> > > > > HiveServer2
> > > > > > > > > Endpoint[1]. The Endpoint will implement the thrift
> interface
> > > > > exposed
> > > > > > > by
> > > > > > > > > the HiveServer2, and users' BI, CLI and other tools based
> on
> > > the
> > > > > > > > > HiveServer2 can also be seamlessly migrated to the Flink
> SQL
> > > > > Gateway.
> > > > > > > > After
> > > > > > > > > the FLIP finishes, the users can have almost the same
> > > experience
> > > > in
> > > > > > the
> > > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the
> > > > > > HiveServer2.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I am glad that you can give some feedback about FLIP-223.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Shengkai
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Martijn Visser <ma...@ververica.com>.
Hi Shengkai,
Thanks for clarifying.
Best regards,
Martijn
On Fri, 6 May 2022 at 08:40, Shengkai Fang <fs...@gmail.com> wrote:
> Hi Martijn.
>
> > So this implementation would not rely in any way on Hive, only on Thrift?
>
> Yes. The dependency is light. We also can just copy the iface file from
> the Hive repo and maintain by ourselves.
>
> Best,
> Shengkai
>
> Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
>
> > Hi Shengkai,
> >
> > > Actually we will only rely on the API in the Hive, which only contains
> > the thrift file and the generated code
> >
> > So this implementation would not rely in any way on Hive, only on Thrift?
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> >
> > On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com> wrote:
> >
> > > Hi, Jark and Martijn
> > >
> > > Thanks for your feedback.
> > >
> > > > Kyuubi provides three ways to configure Hive metastore [1]. Could we
> > > provide similar abilities?
> > >
> > > Yes. I have updated the FLIP about this and it takes some time to
> figure
> > > out how the jdbc driver works. I added the section about how to use the
> > > hive JDBC to configure the session-level catalog.
> > >
> > > > I think we can improve the "HiveServer2 Compatibility" section.
> > >
> > > Yes. I have updated the FLIP and added more details about the
> > > compatibility.
> > >
> > > > Prefer to first complete the discussion and vote on FLIP-91 then
> > discuss
> > > FLIP-223
> > >
> > > Of course. We can wait until the discussion of the FLIP-91 finishes.
> > >
> > > > Maintenance concerns about the hive
> > >
> > > Actually we will only rely on the API in the Hive, which only contains
> > the
> > > thrift file and the generated code[1]. I think it will not influence us
> > to
> > > upgrade the java version.
> > >
> > > [1] https://github.com/apache/hive/tree/master/service-rpc
> > >
> > > Best,
> > > Shengkai
> > >
> > > Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
> > >
> > > > Hi all,
> > > >
> > > > I'm not too familiar with Hive and HiveServer2, but I do have a
> couple
> > of
> > > > questions/concerns:
> > > >
> > > > 1. What is the relationship between this FLIP and FLIP-91? My
> > assumption
> > > > would be that this FLIP (and therefore the HiveServer2)
> implementation
> > > > would need to be integrated in the REST Gateway, is that correct? If
> > so,
> > > I
> > > > would prefer to first complete the discussion and vote on FLIP-91,
> else
> > > > we'll have two moving FLIPs who have a direct relationship with each
> > > other.
> > > >
> > > > 2. While I understand that Hive is important (in the Chinese
> ecosystem,
> > > not
> > > > so much in Europe and the US), I still have maintenance concerns on
> > this
> > > > topic. We know that the current Hive integration isn't exactly ideal
> > and
> > > > requires a lot of work to get in better shape. At the same time, Hive
> > > still
> > > > doesn't support Java 11 while we need (and should, given the premier
> > > > support has ended already) to move away from Java 8.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn Visser
> > > > https://twitter.com/MartijnVisser82
> > > > https://github.com/MartijnVisser
> > > >
> > > >
> > > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com> wrote:
> > > >
> > > > > Thank Shengkai for driving this effort,
> > > > > I think this is an essential addition to Flink Batch.
> > > > >
> > > > > I have some small suggestions:
> > > > > 1) Kyuubi provides three ways to configure Hive metastore [1].
> Could
> > we
> > > > > provide similar abilities?
> > > > > Especially with the JDBC Connection URL, users can visit different
> > Hive
> > > > > metastore server instances.
> > > > >
> > > > > 2) I think we can improve the "HiveServer2 Compatibility" section.
> > > > > We need to figure out two compatibility matrices. One is SQL
> Gateway
> > > with
> > > > > different versions of Hive metastore,
> > > > > and the other is different versions of Hive client (e.g., Hive
> JDBC)
> > > with
> > > > > SQL Gateway. We need to clarify
> > > > > what metastore and client versions we support and how users
> configure
> > > the
> > > > > versions.
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > >
> > >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > > >
> > > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fs...@gmail.com>
> > wrote:
> > > > >
> > > > > > Hi, Jiang.
> > > > > >
> > > > > > Thanks for your feedback!
> > > > > >
> > > > > > > Integrating the Hive ecosystem should not require changing the
> > > > service
> > > > > > interface
> > > > > >
> > > > > > I move the API change to the FLIP-91. But I think it's possible
> we
> > > add
> > > > > more
> > > > > > interfaces to intergrate the new endpoints in the future because
> > > every
> > > > > > endpoints's functionality is different. For example, the REST
> > > endpoint
> > > > > > doen't support to fetch operation-level logs but the hiveserver2
> > > > endpoint
> > > > > > supports. In this case, we need to modify the shared
> GatewayService
> > > to
> > > > > > support the functionality exposed by the new endpint.
> > > > > >
> > > > > > > How to support different Hive versions?
> > > > > >
> > > > > > Do you means to support the different HiveServer2 version? The
> > > > > HiveServer2
> > > > > > uses the version to guarantee the compatibility. During the
> > > > openSession,
> > > > > > the client and server will determine the protocol
> > > > version(minimun(client
> > > > > > version, hiveendpoint version)). After that the client and the
> > server
> > > > > uses
> > > > > > the determined version to communicate. In the HiveServer2
> endpoint,
> > > it
> > > > > > determines how the endpoint deserialize the results and the
> result
> > > > > schema.
> > > > > > I add a section about HiveServer2 compatiblity.
> > > > > >
> > > > > > > Could you please fully provide its definition including input
> > > > > parameters
> > > > > > and the corresponding return value schema?
> > > > > >
> > > > > > Because we implements the interface exposed by the Hive. So I add
> > the
> > > > > file
> > > > > > link to the HiveServer2 interfaces[1], which contains all input
> > > > > parameters
> > > > > > and the results. Considering the file doesn't contain the output
> > for
> > > > the
> > > > > > Operation, I add the output schema for all the supported
> Operation
> > in
> > > > the
> > > > > > FLIP, which is not covered in the link. Hope these can address
> your
> > > > > > question.
> > > > > >
> > > > > > Best,
> > > > > > Shengkai
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:43写道：
> > > > > >
> > > > > > > Hi Shengkai.
> > > > > > >
> > > > > > > Thanks for driving the proposal of HiveServer2 Endpoint
> support.
> > > For
> > > > > the
> > > > > > > "GatewayService API Change", I don't think the motivation for
> > > > > supporting
> > > > > > > HiveServer2 endpoint need to change the GatewayService API, in
> > > other
> > > > > > words,
> > > > > > > integrating the Hive ecosystem should not require changing the
> > > > service
> > > > > > > interface. If you confirm to change GatewayService interface,
> > IMO,
> > > > the
> > > > > > > proposal could be discussed in FLIP-91 because the public
> > > interfaces
> > > > > are
> > > > > > > defined in FLIP-91.
> > > > > > >
> > > > > > > In addtion, how to support different Hive versions and how to
> > > > guarantee
> > > > > > > compatibility is not mentioned in the design. What's the
> behavior
> > > of
> > > > > the
> > > > > > > compatibility?
> > > > > > >
> > > > > > > Finally, for the public interfaces, could you please fully
> > provide
> > > > its
> > > > > > > definition including input parameters and the corresponding
> > return
> > > > > value
> > > > > > > schema?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Nicholas Jiang
> > > > > > >
> > > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > > Hi, Flink developers.
> > > > > > > >
> > > > > > > > I want to start a discussion about the FLIP-223: Support
> > > > HiveServer2
> > > > > > > > Endpoint[1]. The Endpoint will implement the thrift interface
> > > > exposed
> > > > > > by
> > > > > > > > the HiveServer2, and users' BI, CLI and other tools based on
> > the
> > > > > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL
> > > > Gateway.
> > > > > > > After
> > > > > > > > the FLIP finishes, the users can have almost the same
> > experience
> > > in
> > > > > the
> > > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the
> > > > > HiveServer2.
> > > > > > > >
> > > > > > > >
> > > > > > > > I am glad that you can give some feedback about FLIP-223.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Shengkai
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-223: Support HiveServer2 Endpoint
Posted by Shengkai Fang <fs...@gmail.com>.
Hi Martijn.
> So this implementation would not rely in any way on Hive, only on Thrift?
Yes. The dependency is light. We also can just copy the iface file from
the Hive repo and maintain by ourselves.
Best,
Shengkai
Martijn Visser <ma...@apache.org> 于2022年5月4日周三 21:44写道：
> Hi Shengkai,
>
> > Actually we will only rely on the API in the Hive, which only contains
> the thrift file and the generated code
>
> So this implementation would not rely in any way on Hive, only on Thrift?
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
>
> On Fri, 29 Apr 2022 at 05:16, Shengkai Fang <fs...@gmail.com> wrote:
>
> > Hi, Jark and Martijn
> >
> > Thanks for your feedback.
> >
> > > Kyuubi provides three ways to configure Hive metastore [1]. Could we
> > provide similar abilities?
> >
> > Yes. I have updated the FLIP about this and it takes some time to figure
> > out how the jdbc driver works. I added the section about how to use the
> > hive JDBC to configure the session-level catalog.
> >
> > > I think we can improve the "HiveServer2 Compatibility" section.
> >
> > Yes. I have updated the FLIP and added more details about the
> > compatibility.
> >
> > > Prefer to first complete the discussion and vote on FLIP-91 then
> discuss
> > FLIP-223
> >
> > Of course. We can wait until the discussion of the FLIP-91 finishes.
> >
> > > Maintenance concerns about the hive
> >
> > Actually we will only rely on the API in the Hive, which only contains
> the
> > thrift file and the generated code[1]. I think it will not influence us
> to
> > upgrade the java version.
> >
> > [1] https://github.com/apache/hive/tree/master/service-rpc
> >
> > Best,
> > Shengkai
> >
> > Martijn Visser <ma...@apache.org> 于2022年4月26日周二 20:44写道：
> >
> > > Hi all,
> > >
> > > I'm not too familiar with Hive and HiveServer2, but I do have a couple
> of
> > > questions/concerns:
> > >
> > > 1. What is the relationship between this FLIP and FLIP-91? My
> assumption
> > > would be that this FLIP (and therefore the HiveServer2) implementation
> > > would need to be integrated in the REST Gateway, is that correct? If
> so,
> > I
> > > would prefer to first complete the discussion and vote on FLIP-91, else
> > > we'll have two moving FLIPs who have a direct relationship with each
> > other.
> > >
> > > 2. While I understand that Hive is important (in the Chinese ecosystem,
> > not
> > > so much in Europe and the US), I still have maintenance concerns on
> this
> > > topic. We know that the current Hive integration isn't exactly ideal
> and
> > > requires a lot of work to get in better shape. At the same time, Hive
> > still
> > > doesn't support Java 11 while we need (and should, given the premier
> > > support has ended already) to move away from Java 8.
> > >
> > > Best regards,
> > >
> > > Martijn Visser
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
> > >
> > >
> > > On Mon, 25 Apr 2022 at 12:13, Jark Wu <im...@gmail.com> wrote:
> > >
> > > > Thank Shengkai for driving this effort,
> > > > I think this is an essential addition to Flink Batch.
> > > >
> > > > I have some small suggestions:
> > > > 1) Kyuubi provides three ways to configure Hive metastore [1]. Could
> we
> > > > provide similar abilities?
> > > > Especially with the JDBC Connection URL, users can visit different
> Hive
> > > > metastore server instances.
> > > >
> > > > 2) I think we can improve the "HiveServer2 Compatibility" section.
> > > > We need to figure out two compatibility matrices. One is SQL Gateway
> > with
> > > > different versions of Hive metastore,
> > > > and the other is different versions of Hive client (e.g., Hive JDBC)
> > with
> > > > SQL Gateway. We need to clarify
> > > > what metastore and client versions we support and how users configure
> > the
> > > > versions.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> https://kyuubi.apache.org/docs/r1.3.1-incubating/deployment/hive_metastore.html#activate-configurations
> > > >
> > > > On Sun, 24 Apr 2022 at 15:02, Shengkai Fang <fs...@gmail.com>
> wrote:
> > > >
> > > > > Hi, Jiang.
> > > > >
> > > > > Thanks for your feedback!
> > > > >
> > > > > > Integrating the Hive ecosystem should not require changing the
> > > service
> > > > > interface
> > > > >
> > > > > I move the API change to the FLIP-91. But I think it's possible we
> > add
> > > > more
> > > > > interfaces to intergrate the new endpoints in the future because
> > every
> > > > > endpoints's functionality is different. For example, the REST
> > endpoint
> > > > > doen't support to fetch operation-level logs but the hiveserver2
> > > endpoint
> > > > > supports. In this case, we need to modify the shared GatewayService
> > to
> > > > > support the functionality exposed by the new endpint.
> > > > >
> > > > > > How to support different Hive versions?
> > > > >
> > > > > Do you means to support the different HiveServer2 version? The
> > > > HiveServer2
> > > > > uses the version to guarantee the compatibility. During the
> > > openSession,
> > > > > the client and server will determine the protocol
> > > version(minimun(client
> > > > > version, hiveendpoint version)). After that the client and the
> server
> > > > uses
> > > > > the determined version to communicate. In the HiveServer2 endpoint,
> > it
> > > > > determines how the endpoint deserialize the results and the result
> > > > schema.
> > > > > I add a section about HiveServer2 compatiblity.
> > > > >
> > > > > > Could you please fully provide its definition including input
> > > > parameters
> > > > > and the corresponding return value schema?
> > > > >
> > > > > Because we implements the interface exposed by the Hive. So I add
> the
> > > > file
> > > > > link to the HiveServer2 interfaces[1], which contains all input
> > > > parameters
> > > > > and the results. Considering the file doesn't contain the output
> for
> > > the
> > > > > Operation, I add the output schema for all the supported Operation
> in
> > > the
> > > > > FLIP, which is not covered in the link. Hope these can address your
> > > > > question.
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/hive/blob/branch-2.3/service-rpc/if/TCLIService.thrift#L1227
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Nicholas Jiang <ni...@apache.org> 于2022年4月22日周五 16:43写道：
> > > > >
> > > > > > Hi Shengkai.
> > > > > >
> > > > > > Thanks for driving the proposal of HiveServer2 Endpoint support.
> > For
> > > > the
> > > > > > "GatewayService API Change", I don't think the motivation for
> > > > supporting
> > > > > > HiveServer2 endpoint need to change the GatewayService API, in
> > other
> > > > > words,
> > > > > > integrating the Hive ecosystem should not require changing the
> > > service
> > > > > > interface. If you confirm to change GatewayService interface,
> IMO,
> > > the
> > > > > > proposal could be discussed in FLIP-91 because the public
> > interfaces
> > > > are
> > > > > > defined in FLIP-91.
> > > > > >
> > > > > > In addtion, how to support different Hive versions and how to
> > > guarantee
> > > > > > compatibility is not mentioned in the design. What's the behavior
> > of
> > > > the
> > > > > > compatibility?
> > > > > >
> > > > > > Finally, for the public interfaces, could you please fully
> provide
> > > its
> > > > > > definition including input parameters and the corresponding
> return
> > > > value
> > > > > > schema?
> > > > > >
> > > > > > Thanks,
> > > > > > Nicholas Jiang
> > > > > >
> > > > > > On 2022/04/21 06:45:13 Shengkai Fang wrote:
> > > > > > > Hi, Flink developers.
> > > > > > >
> > > > > > > I want to start a discussion about the FLIP-223: Support
> > > HiveServer2
> > > > > > > Endpoint[1]. The Endpoint will implement the thrift interface
> > > exposed
> > > > > by
> > > > > > > the HiveServer2, and users' BI, CLI and other tools based on
> the
> > > > > > > HiveServer2 can also be seamlessly migrated to the Flink SQL
> > > Gateway.
> > > > > > After
> > > > > > > the FLIP finishes, the users can have almost the same
> experience
> > in
> > > > the
> > > > > > > Flink SQL Gateway with the HiveServer2 endpoint as in the
> > > > HiveServer2.
> > > > > > >
> > > > > > >
> > > > > > > I am glad that you can give some feedback about FLIP-223.
> > > > > > >
> > > > > > > Best,
> > > > > > > Shengkai
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-223+Support+HiveServer2+Endpoint
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>