Flink kicks off Calcite integration

[Julian Hyde <jh...@apache.org>]

First of all, apologies for cross-posting. If you reply, please remove
dev@flink and reply to dev@calcite only.

Calcite contributors,

Fabian Hueske has sent an email to dev@flink [1] on their plans to add
SQL and streaming SQL support to Flink using Calcite, and also
included a design document [2].

We should be proud that they are putting their faith in us. I know
that Calcite a sound platform for them to build on, but I also know
that we will have to help them out, answering questions, adding
features, and fixing bugs. So, please help them any way you can. The
fact that we integrate with lots of projects (Drill, Hive, Phoenix,
Kylin, Samza, Storm and now Flink) makes us a stronger platform for
everyone.

Do any of you want to get more involved in that effort? It would be
great if one or two Calcite contributors were monitoring the Flink dev
list, and were able to step in and help out. (I'm already maxed out
with integrations: I subscribe to the Drill, Kylin and Samza dev
lists, and have a daily stand up with Hive devs at Hortonworks.)

Fabian and other Flink contributors,

In future, feel free to CC: the Calcite dev list when you send emails
about Calcite integration to the Flink list. I wouldn't consider it
cross-posting. If there are interesting issues raised, folks on the
Calcite side will subscribe to the Flink list (or watch the applicable
Flink JIRA case). We want to help you out any way we can.

Julian

[1] https://mail-archives.apache.org/mod_mbox/flink-dev/201601.mbox/%3CCAAdrtT04o6tU%2BDHM4632v_8p7QxT7ErFWicpuzFUtfaDMOK9FQ%40mail.gmail.com%3E

[2]

https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit?usp=sharing

Re: Flink kicks off Calcite integration

[Jesus Camacho Rodriguez <jc...@hortonworks.com>]

Hi Julian,

This is indeed very interesting for the project, the wider usage the better.

I will try to get more involved in case they have questions about Calcite, as I already know the Flink guys.
Further, I think we could benefit on the Hive side from the work that they will be doing, specially on the optimizer side of Calcite.

I will take a look at their design doc.

--
Jesús



On 1/8/16, 8:06 PM, "Julian Hyde" <jh...@apache.org> wrote:

>First of all, apologies for cross-posting. If you reply, please remove
>dev@flink and reply to dev@calcite only.
>
>Calcite contributors,
>
>Fabian Hueske has sent an email to dev@flink [1] on their plans to add
>SQL and streaming SQL support to Flink using Calcite, and also
>included a design document [2].
>
>We should be proud that they are putting their faith in us. I know
>that Calcite a sound platform for them to build on, but I also know
>that we will have to help them out, answering questions, adding
>features, and fixing bugs. So, please help them any way you can. The
>fact that we integrate with lots of projects (Drill, Hive, Phoenix,
>Kylin, Samza, Storm and now Flink) makes us a stronger platform for
>everyone.
>
>Do any of you want to get more involved in that effort? It would be
>great if one or two Calcite contributors were monitoring the Flink dev
>list, and were able to step in and help out. (I'm already maxed out
>with integrations: I subscribe to the Drill, Kylin and Samza dev
>lists, and have a daily stand up with Hive devs at Hortonworks.)
>
>Fabian and other Flink contributors,
>
>In future, feel free to CC: the Calcite dev list when you send emails
>about Calcite integration to the Flink list. I wouldn't consider it
>cross-posting. If there are interesting issues raised, folks on the
>Calcite side will subscribe to the Flink list (or watch the applicable
>Flink JIRA case). We want to help you out any way we can.
>
>Julian
>
>[1] https://mail-archives.apache.org/mod_mbox/flink-dev/201601.mbox/%3CCAAdrtT04o6tU%2BDHM4632v_8p7QxT7ErFWicpuzFUtfaDMOK9FQ%40mail.gmail.com%3E
>
>[2]
>
>https://docs.google.com/document/d/1TLayJNOTBle_-m1rQfgA6Ouj1oYsfqRjPcp1h2TVqdI/edit?usp=sharing
>

Re: Flink kicks off Calcite integration

[Fabian Hueske <fh...@gmail.com>]

Hi everybody,

First of all, thanks Julian for reaching out to us and Jesus for offering
your help.
I am excited that the Flink community is finally starting to add SQL
support for batch data sets and data streams to Flink.
I believe Calcite will serve as an excellent component in our architecture.

As described in the design document, the first step will be to put Flink's
existing Table API on top of Calcite. At the moment, the Table API only
supports a very basic set of relational operators (selection, projection,
equi-inner joins, grouping, some basic aggregation functions). All of that
is very well covered by Calcite and I think we mainly need a few custom
translation rules to generate a physical plan that represents a Flink
DataSet program.

Once this first step is completed, we will start to work towards support
for continuous queries over streaming data.
I expect that we will have some questions when we reach this point.

Looking forward to get this started an thanks again,
Fabian