You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by 953396112 <13...@qq.com> on 2020/12/12 15:36:46 UTC
回复： Using Calcite at LinkedIn
Hi Walaa and Tao:
I am very happy to see your sharing. Our team is also working on SQL rewriting, analysis and optimization. Using Calcite's materialized view recognition capabilities to speed up user queries, many materialized view recognition algorithms[1] and normalization algorithms[2] have been implemented, and a lot of work has been done in preprocessing available materialized views, which has excellent performance. Welcome to discuss issues related to materialized view identification.
[1] https://github.com/apache/calcite/pull/2094
[2] https://github.com/apache/calcite/pull/2262
Regards!
Zhaohui Xu
------------------ 原始邮件 ------------------
发件人: "dev" <taojiatao@gmail.com>;
发送时间: 2020年12月12日(星期六) 晚上9:23
收件人: "dev"<dev@calcite.apache.org>;
主题: Re: Using Calcite at LinkedIn
Hi Walaa
Very happy to see this, our team basically do the same thing, a unified SQL
layer:
1. Spark: RelNode -> Spark DataFrame plan
2. Presto: RelNode -> In string SQL
3. Clickhouse: RelNode -> Serialized RelNode
4. Flink -> TBD(with datastream API or table API)
I do point 1 both in my previous company and current company, maybe I can
participate in this part: analyze and translate Spark Catalyst plans.
Regards!
Aron Tao
Walaa Eldin Moustafa <wa.moustafa@gmail.com> 于2020年12月12日周六 上午5:34写道：
> Hi Calcite community,
>
> I wanted to share a recently published LinkedIn's blog series article [1]
> on how Calcite helps us build a smarter data lake using Coral [2]. Hope you
> find it interesting. Also, if you want to discuss with our team and the
> data lake + Calcite community, please feel free to join our Coral Slack
> workspace [3].
>
> [1] https://engineering.linkedin.com/blog/2020/coral
> [2] https://github.com/linkedin/coral
> [3]
>
> https://join.slack.com/t/coral-sql/shared_invite/zt-j9jw5idg-mkt3fjA~wgoUEMXXZqMr6g
>
> Thanks,
> Walaa.
>
Re: Using Calcite at LinkedIn
Posted by Walaa Eldin Moustafa <wa...@gmail.com>.
Hi Zhaohui,
Thanks for sharing. Materialized views are crucial to performance
optimization in the data lake. We mentioned that in the blog post too as a
future work. Would love to discuss this further.
Thanks,
Walaa.
On Sat, Dec 12, 2020 at 11:49 AM 953396112 <13...@qq.com> wrote:
> Hi Walaa and Tao:
> I am very happy to see your sharing. Our team is also
> working on SQL rewriting, analysis and optimization. Using Calcite's
> materialized view recognition capabilities to speed up user queries, many
> materialized view recognition algorithms[1] and normalization algorithms[2]
> have been implemented, and a lot of work has been done in preprocessing
> available materialized views, which has excellent performance. Welcome to
> discuss issues related to materialized view identification.
>
>
> [1] https://github.com/apache/calcite/pull/2094
> [2] https://github.com/apache/calcite/pull/2262
>
>
> Regards!
> Zhaohui Xu
>
>
> ------------------ 原始邮件 ------------------
> 发件人:
> "dev"
> <
> taojiatao@gmail.com>;
> 发送时间: 2020年12月12日(星期六) 晚上9:23
> 收件人: "dev"<dev@calcite.apache.org>;
>
> 主题: Re: Using Calcite at LinkedIn
>
>
>
> Hi Walaa
> Very happy to see this, our team basically do the same thing, a unified SQL
> layer:
> 1. Spark: RelNode -> Spark DataFrame plan
> 2. Presto: RelNode -> In string SQL
> 3. Clickhouse: RelNode -> Serialized RelNode
> 4. Flink -> TBD(with datastream API or table API)
>
>
> I do point 1 both in my previous company and current company, maybe I can
> participate in this part: analyze and translate Spark Catalyst plans.
>
>
> Regards!
>
> Aron Tao
>
>
> Walaa Eldin Moustafa <wa.moustafa@gmail.com> 于2020年12月12日周六 上午5:34写道：
>
> > Hi Calcite community,
> >
> > I wanted to share a recently published LinkedIn's blog series article
> [1]
> > on how Calcite helps us build a smarter data lake using Coral [2].
> Hope you
> > find it interesting. Also, if you want to discuss with our team and
> the
> > data lake + Calcite community, please feel free to join our Coral
> Slack
> > workspace [3].
> >
> > [1] https://engineering.linkedin.com/blog/2020/coral
> > [2] https://github.com/linkedin/coral
> > [3]
> >
> >
> https://join.slack.com/t/coral-sql/shared_invite/zt-j9jw5idg-mkt3fjA~wgoUEMXXZqMr6g
> >
> <https://join.slack.com/t/coral-sql/shared_invite/zt-j9jw5idg-mkt3fjA~wgoUEMXXZqMr6g>>
> ;
> > Thanks,
> > Walaa.
> >