You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by 罗宇侠 <yu...@qq.com.INVALID> on 2022/03/25 12:19:37 UTC
FLIP-216 Decouple Hive connector with Flink planner
Hi, everyone
I would like to open a discussion about decoupling Hive connector with Flink table planner. It's a follow-up discussion after Hive syntax discussion[1], but only focus on how to decouple Hive connector. The origin doc is here[2], from which you can see the details work and heated discussion about exposing Calcite API or reuse Operation tree to decouple.
I have created FLIP-216: Decouple Hive connector with Flink planner[3] for it.
Thanks and looking forward to a lively discussion!
[1] https://lists.apache.org/thread/2w046dwl46tf2wy750gzmt0qrcz17z8t
[2] https://docs.google.com/document/d/1LMQ_mWfB_mkYkEBCUa2DgCO2YdtiZV7YRs2mpXyjdP4/edit?usp=sharing
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A+Decouple+Hive+connector+with+Flink+planner
Best regards,
Yuxia
Re: FLIP-216 Decouple Hive connector with Flink planner
Posted by Francesco Guardiani <fr...@ververica.com>.
As there was already a discussion in the doc, I'll just summarize my
opinions here on the proposed execution of this FLIP.
I think we should rather avoid exposing internal details, which I consider
Calcite to be part of, but rather reuse what we already have to define an
AST from Table API, which is what I'll refer in this mail as Operation tree.
First of all, the reason I think this FLIP is not a good idea is that it
proposes is to expose types out of our control, so an API we cannot control
and we may realistically never be able to stabilize. A Calcite bump in the
table project is already pretty hard today, as shown by tasks like that
https://github.com/apache/flink/pull/13577. This will make them even
harder. Essentially it will couple us to Calcite even more, and create a
different but still big maintenance/complexity burden we would like to get
rid of with this FLIP.
There are also some technical aspects that seem to me a bit overlooked here:
* What about Scala? Is flink-table-planner-spi going to be a scala module
with the related suffix? Because I see you want to expose a couple of types
which we have implemented with Scala right now, and making this module
Scala dependent makes even more complicated shipping both modules that use
it and flink-table-planner-loader.
* Are you sure exposing the Calcite interfaces is going to be enough? Don't
you also require some instance specific methods? E.g.
FlinkTypeFactory#toLogicalType? What if at some point you need to expose
something like FlinkTypeFactory? How do you plan to support it and
stabilize it in the long term?
Now let me talk a bit about the Operation tree. For who doesn't know what
it is, it's the pure Flink AST for defining DML, used for converting the
Table API DSL to an AST the planner can manipulate. Essentially, it's our
own version of the RelNode tree/RexNode tree. This operation tree can be
used already by Hive, without any API changes on Table side. You just need
a downcast of TableEnvironmentImpl to use getPlanner() and use
Planner#translate, or alternatively you can add getPlanner to
TableEnvironmentInternal directly. From what I've seen about your use case,
and please correct me if I'm wrong, you can implement your SQL -> Operation
tree layer without substantial changes on both sides.
The reason why I think this is a better idea, rather than exposing Calcite
and RelNodes directly, is:
* Aforementioned downsides of exposing Calcite
* It doesn't require a new API to get you started with it
* Doesn't add complexity on planner side, just removes it from the existing
coupling with hive
* Letting another project use the Operation tree will harden it, make it
more stable and eventually lead to become public
The last point in particular is extremely interesting for the future of the
project, as having a stable public Operation tree will allow people to
implement other relational based APIs on top of Flink SQL, or manipulate
the AST to define new semantics, or even more crazy things we can't think
of right now, leading to a broader bigger and more diverse ecosystem. Which
is exactly what Hive is doing right now at the end of the day, define a new
relational API on top of the Flink Table planner functionalities.
On Mon, Mar 28, 2022 at 10:57 AM 罗宇侠(莫辞)
<lu...@alibaba-inc.com.invalid> wrote:
> Sorry for this email, seems there's some format issue in my email client.
> Just ignore it for it's a duplicate of [DISCUSS] FLIP-216 Decouple Hive
> connector with Flink planner [1]
>
>
> [1] https://lists.apache.org/thread/6xg33nxrnow5zy7xwqk5nwp00h9gcsbc
>
> Best regards,
> Yuxia------------------------------------------------------------------
> 发件人：罗宇侠<yu...@qq.com.INVALID>
> 日 期：2022年03月25日 20:19:37
> 收件人：dev<de...@flink.apache.org>
> 主 题：FLIP-216 Decouple Hive connector with Flink planner
>
> Hi, everyone
>
>
>
>
> I would like to open a discussion about decoupling Hive connector with Flink table planner.
> It's a follow-up discussion after Hive syntax discussion[1], but only focus on how to decouple Hive connector. The origin doc is here[2], from which you can see the details work and heated discussion about exposing Calcite API or reuse Operation tree to decouple.
>
> I have created FLIP-216: Decouple Hive connector with Flink planner[3] for it.
>
>
>
>
>
> Thanks and looking forward to a lively discussion!
>
>
>
> [1] https://lists.apache.org/thread/2w046dwl46tf2wy750gzmt0qrcz17z8t
>
> [2]
> https://docs.google.com/document/d/1LMQ_mWfB_mkYkEBCUa2DgCO2YdtiZV7YRs2mpXyjdP4/edit?usp=sharing
>
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A+Decouple+Hive+connector+with+Flink+planner
>
>
>
>
> Best regards,
> Yuxia
>
>
>
>
回复：FLIP-216 Decouple Hive connector with Flink planner
Posted by "罗宇侠(莫辞)" <lu...@alibaba-inc.com.INVALID>.
Sorry for this email, seems there's some format issue in my email client. Just ignore it for it's a duplicate of [DISCUSS] FLIP-216 Decouple Hive connector with Flink planner [1]
[1] https://lists.apache.org/thread/6xg33nxrnow5zy7xwqk5nwp00h9gcsbc
Best regards,
Yuxia------------------------------------------------------------------
发件人：罗宇侠<yu...@qq.com.INVALID>
日 期：2022年03月25日 20:19:37
收件人：dev<de...@flink.apache.org>
主 题：FLIP-216 Decouple Hive connector with Flink planner
Hi, everyone
I would like to open a discussion about decoupling Hive connector with Flink table planner. It's a follow-up discussion after Hive syntax discussion[1], but only focus on how to decouple Hive connector. The origin doc is here[2], from which you can see the details work and heated discussion about exposing Calcite API or reuse Operation tree to decouple.
I have created FLIP-216: Decouple Hive connector with Flink planner[3] for it.
Thanks and looking forward to a lively discussion!
[1] https://lists.apache.org/thread/2w046dwl46tf2wy750gzmt0qrcz17z8t
[2] https://docs.google.com/document/d/1LMQ_mWfB_mkYkEBCUa2DgCO2YdtiZV7YRs2mpXyjdP4/edit?usp=sharing
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A+Decouple+Hive+connector+with+Flink+planner
Best regards,
Yuxia