You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by LuNing Wang <wa...@gmail.com> on 2022/09/01 02:25:06 UTC
Re: [DISCUSS] Introduce Iceberg management REST API into Kyuubi
I find only one procedure[1] in the Trino iceberg connector. We can
encapsulate it in REST API. We can generate different SQL according to
different engines. When the engine is Spark, we generate Spark procedure,
when the engine is Trino generate Trino ones.
[1]:
https://trino.io/docs/current/connector/iceberg.html#rolling-back-to-a-previous-snapshot
Kent Yao <ya...@apache.org> 于2022年8月31日周三 17:21写道：
> are you interested in extending our rest APIs to support trino ones?
>
> LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 17:07写道：
> >
> > I have tried to submit Iceberg Rewrite Data SQL to
> > SessionsResource/executeStatement. It works and successfully get result
> > success.
> >
> > Kent Yao <ya...@apache.org> 于2022年8月31日周三 14:41写道：
> >
> > > Oh, my bad. It's in the SessionsResource
> > >
> > > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 14:35写道：
> > > >
> > > > I can't find any executeStatement method in OperationResouces. Shall
> we
> > > > implement OperationResouces/executeStatement and then use
> > > > OperationResouces to implement Iceberg REST API?
> > > > OperationResouces/executeStatement will auto open and close a
> session to
> > > > execute a SQL statement.
> > > >
> > > > Kent Yao <ya...@apache.org> 于2022年8月23日周二 15:11写道：
> > > >
> > > > > Do we already support this via OperationResouces/executeStatement?
> > > > >
> > > > > On 2022/08/19 08:44:35 LuNing Wang wrote:
> > > > > > Hi Kyuubi community,
> > > > > >
> > > > > > Since the iceberg has some spark procedures
> > > > > > <
> > > > >
> > >
> https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > >[1]
> > > > > > to manage the iceberg table and KPIP-1
> > > > > > <https://github.com/apache/incubator-kyuubi/discussions/917>[2]
> has
> > > > > > introduced REST API into the Kyuubi. So we can add some iceberg
> > > > > management
> > > > > > REST API into Kyuubi. If so, users can only use REST API to
> manage
> > > the
> > > > > > iceberg table via Kyuubi, and some users also can config iceberg
> > > > > management
> > > > > > jobs in their big-data web platform via this REST API.
> > > > > >
> > > > > > Preliminary draft of the scheme:
> > > > > >
> > > > > > https://github.com/apache/incubator-kyuubi/discussions/3258
> > > > > >
> > > > > > A simple demo:
> > > > > >
> > > > > >
> https://github.com/deadwind4/incubator-kyuubi/tree/iceberg_rewrite
> > > > > >
> > > > > > best regards,
> > > > > >
> > > > > > Luning Wang
> > > > > >
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > >
> > >
> https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > >
> > > > > > [2]: https://github.com/apache/incubator-kyuubi/discussions/917
> > > > > >
> > > > >
> > >
>
Re: [DISCUSS] Introduce Iceberg management REST API into Kyuubi
Posted by Kent Yao <ya...@apache.org>.
thanks LuNing~
On 2022/09/02 09:42:00 LuNing Wang wrote:
> I would like to support Trino REST frontend service, I will draft a design
> doc, and then create an umbrella issue.
>
> Kent Yao <ya...@apache.org> 于2022年9月1日周四 11:10写道：
>
> > I mean we support trino frontend service[1] for Kyuubi server which
> > may be out of the scope of the original discussion and then expanding
> > it to support procedure calls if necessary
> >
> >
> > 1 https://trino.io/docs/current/develop/client-protocol.html
> >
> > LuNing Wang <wa...@gmail.com> 于2022年9月1日周四 10:25写道：
> > >
> > > I find only one procedure[1] in the Trino iceberg connector. We can
> > > encapsulate it in REST API. We can generate different SQL according to
> > > different engines. When the engine is Spark, we generate Spark procedure,
> > > when the engine is Trino generate Trino ones.
> > >
> > > [1]:
> > >
> > https://trino.io/docs/current/connector/iceberg.html#rolling-back-to-a-previous-snapshot
> > >
> > > Kent Yao <ya...@apache.org> 于2022年8月31日周三 17:21写道：
> > >
> > > > are you interested in extending our rest APIs to support trino ones?
> > > >
> > > > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 17:07写道：
> > > > >
> > > > > I have tried to submit Iceberg Rewrite Data SQL to
> > > > > SessionsResource/executeStatement. It works and successfully get
> > result
> > > > > success.
> > > > >
> > > > > Kent Yao <ya...@apache.org> 于2022年8月31日周三 14:41写道：
> > > > >
> > > > > > Oh, my bad. It's in the SessionsResource
> > > > > >
> > > > > > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 14:35写道：
> > > > > > >
> > > > > > > I can't find any executeStatement method in OperationResouces.
> > Shall
> > > > we
> > > > > > > implement OperationResouces/executeStatement and then use
> > > > > > > OperationResouces to implement Iceberg REST API?
> > > > > > > OperationResouces/executeStatement will auto open and close a
> > > > session to
> > > > > > > execute a SQL statement.
> > > > > > >
> > > > > > > Kent Yao <ya...@apache.org> 于2022年8月23日周二 15:11写道：
> > > > > > >
> > > > > > > > Do we already support this via
> > OperationResouces/executeStatement?
> > > > > > > >
> > > > > > > > On 2022/08/19 08:44:35 LuNing Wang wrote:
> > > > > > > > > Hi Kyuubi community,
> > > > > > > > >
> > > > > > > > > Since the iceberg has some spark procedures
> > > > > > > > > <
> > > > > > > >
> > > > > >
> > > >
> > https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > > > > >[1]
> > > > > > > > > to manage the iceberg table and KPIP-1
> > > > > > > > > <https://github.com/apache/incubator-kyuubi/discussions/917
> > >[2]
> > > > has
> > > > > > > > > introduced REST API into the Kyuubi. So we can add some
> > iceberg
> > > > > > > > management
> > > > > > > > > REST API into Kyuubi. If so, users can only use REST API to
> > > > manage
> > > > > > the
> > > > > > > > > iceberg table via Kyuubi, and some users also can config
> > iceberg
> > > > > > > > management
> > > > > > > > > jobs in their big-data web platform via this REST API.
> > > > > > > > >
> > > > > > > > > Preliminary draft of the scheme:
> > > > > > > > >
> > > > > > > > > https://github.com/apache/incubator-kyuubi/discussions/3258
> > > > > > > > >
> > > > > > > > > A simple demo:
> > > > > > > > >
> > > > > > > > >
> > > > https://github.com/deadwind4/incubator-kyuubi/tree/iceberg_rewrite
> > > > > > > > >
> > > > > > > > > best regards,
> > > > > > > > >
> > > > > > > > > Luning Wang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1]:
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > > > > >
> > > > > > > > > [2]:
> > https://github.com/apache/incubator-kyuubi/discussions/917
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >
>
Re: [DISCUSS] Introduce Iceberg management REST API into Kyuubi
Posted by LuNing Wang <wa...@gmail.com>.
I would like to support Trino REST frontend service, I will draft a design
doc, and then create an umbrella issue.
Kent Yao <ya...@apache.org> 于2022年9月1日周四 11:10写道：
> I mean we support trino frontend service[1] for Kyuubi server which
> may be out of the scope of the original discussion and then expanding
> it to support procedure calls if necessary
>
>
> 1 https://trino.io/docs/current/develop/client-protocol.html
>
> LuNing Wang <wa...@gmail.com> 于2022年9月1日周四 10:25写道：
> >
> > I find only one procedure[1] in the Trino iceberg connector. We can
> > encapsulate it in REST API. We can generate different SQL according to
> > different engines. When the engine is Spark, we generate Spark procedure,
> > when the engine is Trino generate Trino ones.
> >
> > [1]:
> >
> https://trino.io/docs/current/connector/iceberg.html#rolling-back-to-a-previous-snapshot
> >
> > Kent Yao <ya...@apache.org> 于2022年8月31日周三 17:21写道：
> >
> > > are you interested in extending our rest APIs to support trino ones?
> > >
> > > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 17:07写道：
> > > >
> > > > I have tried to submit Iceberg Rewrite Data SQL to
> > > > SessionsResource/executeStatement. It works and successfully get
> result
> > > > success.
> > > >
> > > > Kent Yao <ya...@apache.org> 于2022年8月31日周三 14:41写道：
> > > >
> > > > > Oh, my bad. It's in the SessionsResource
> > > > >
> > > > > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 14:35写道：
> > > > > >
> > > > > > I can't find any executeStatement method in OperationResouces.
> Shall
> > > we
> > > > > > implement OperationResouces/executeStatement and then use
> > > > > > OperationResouces to implement Iceberg REST API?
> > > > > > OperationResouces/executeStatement will auto open and close a
> > > session to
> > > > > > execute a SQL statement.
> > > > > >
> > > > > > Kent Yao <ya...@apache.org> 于2022年8月23日周二 15:11写道：
> > > > > >
> > > > > > > Do we already support this via
> OperationResouces/executeStatement?
> > > > > > >
> > > > > > > On 2022/08/19 08:44:35 LuNing Wang wrote:
> > > > > > > > Hi Kyuubi community,
> > > > > > > >
> > > > > > > > Since the iceberg has some spark procedures
> > > > > > > > <
> > > > > > >
> > > > >
> > >
> https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > > > >[1]
> > > > > > > > to manage the iceberg table and KPIP-1
> > > > > > > > <https://github.com/apache/incubator-kyuubi/discussions/917
> >[2]
> > > has
> > > > > > > > introduced REST API into the Kyuubi. So we can add some
> iceberg
> > > > > > > management
> > > > > > > > REST API into Kyuubi. If so, users can only use REST API to
> > > manage
> > > > > the
> > > > > > > > iceberg table via Kyuubi, and some users also can config
> iceberg
> > > > > > > management
> > > > > > > > jobs in their big-data web platform via this REST API.
> > > > > > > >
> > > > > > > > Preliminary draft of the scheme:
> > > > > > > >
> > > > > > > > https://github.com/apache/incubator-kyuubi/discussions/3258
> > > > > > > >
> > > > > > > > A simple demo:
> > > > > > > >
> > > > > > > >
> > > https://github.com/deadwind4/incubator-kyuubi/tree/iceberg_rewrite
> > > > > > > >
> > > > > > > > best regards,
> > > > > > > >
> > > > > > > > Luning Wang
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]:
> > > > > > > >
> > > > > > >
> > > > >
> > >
> https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > > > >
> > > > > > > > [2]:
> https://github.com/apache/incubator-kyuubi/discussions/917
> > > > > > > >
> > > > > > >
> > > > >
> > >
>
Re: [DISCUSS] Introduce Iceberg management REST API into Kyuubi
Posted by Kent Yao <ya...@apache.org>.
I mean we support trino frontend service[1] for Kyuubi server which
may be out of the scope of the original discussion and then expanding
it to support procedure calls if necessary
1 https://trino.io/docs/current/develop/client-protocol.html
LuNing Wang <wa...@gmail.com> 于2022年9月1日周四 10:25写道：
>
> I find only one procedure[1] in the Trino iceberg connector. We can
> encapsulate it in REST API. We can generate different SQL according to
> different engines. When the engine is Spark, we generate Spark procedure,
> when the engine is Trino generate Trino ones.
>
> [1]:
> https://trino.io/docs/current/connector/iceberg.html#rolling-back-to-a-previous-snapshot
>
> Kent Yao <ya...@apache.org> 于2022年8月31日周三 17:21写道：
>
> > are you interested in extending our rest APIs to support trino ones?
> >
> > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 17:07写道：
> > >
> > > I have tried to submit Iceberg Rewrite Data SQL to
> > > SessionsResource/executeStatement. It works and successfully get result
> > > success.
> > >
> > > Kent Yao <ya...@apache.org> 于2022年8月31日周三 14:41写道：
> > >
> > > > Oh, my bad. It's in the SessionsResource
> > > >
> > > > LuNing Wang <wa...@gmail.com> 于2022年8月31日周三 14:35写道：
> > > > >
> > > > > I can't find any executeStatement method in OperationResouces. Shall
> > we
> > > > > implement OperationResouces/executeStatement and then use
> > > > > OperationResouces to implement Iceberg REST API?
> > > > > OperationResouces/executeStatement will auto open and close a
> > session to
> > > > > execute a SQL statement.
> > > > >
> > > > > Kent Yao <ya...@apache.org> 于2022年8月23日周二 15:11写道：
> > > > >
> > > > > > Do we already support this via OperationResouces/executeStatement?
> > > > > >
> > > > > > On 2022/08/19 08:44:35 LuNing Wang wrote:
> > > > > > > Hi Kyuubi community,
> > > > > > >
> > > > > > > Since the iceberg has some spark procedures
> > > > > > > <
> > > > > >
> > > >
> > https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > > >[1]
> > > > > > > to manage the iceberg table and KPIP-1
> > > > > > > <https://github.com/apache/incubator-kyuubi/discussions/917>[2]
> > has
> > > > > > > introduced REST API into the Kyuubi. So we can add some iceberg
> > > > > > management
> > > > > > > REST API into Kyuubi. If so, users can only use REST API to
> > manage
> > > > the
> > > > > > > iceberg table via Kyuubi, and some users also can config iceberg
> > > > > > management
> > > > > > > jobs in their big-data web platform via this REST API.
> > > > > > >
> > > > > > > Preliminary draft of the scheme:
> > > > > > >
> > > > > > > https://github.com/apache/incubator-kyuubi/discussions/3258
> > > > > > >
> > > > > > > A simple demo:
> > > > > > >
> > > > > > >
> > https://github.com/deadwind4/incubator-kyuubi/tree/iceberg_rewrite
> > > > > > >
> > > > > > > best regards,
> > > > > > >
> > > > > > > Luning Wang
> > > > > > >
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > >
> > > >
> > https://iceberg.apache.org/docs/latest/spark-procedures/#spark-procedures
> > > > > > >
> > > > > > > [2]: https://github.com/apache/incubator-kyuubi/discussions/917
> > > > > > >
> > > > > >
> > > >
> >