You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@calcite.apache.org by Khai Tran <kh...@linkedin.com> on 2019/06/07 18:29:31 UTC

Large contribution request: convert Pig Latin scripts into Calcite logical plans and SQL statements

Hi Calcite devs,
I'm Khai Tran from LinkedIn. We have a Calcite internal repo at LinkedIn where we developed a tool to convert any Pig Latin scripts into Calcite logical plans. We used the tool to successfully convert ~1000 Pig scripts written at LinkedIn to Calcite logical plans and would like to contribute it back to the community.

Details of changes we made can be found here: https://issues.apache.org/jira/browse/CALCITE-3122

We used this tool as a part of this batch-streaming converge solution described in this blog post: https://engineering.linkedin.com/blog/2019/01/bridging-offline-and-nearline-computations-with-apache-calcite

We are also considering open source the other part of the solution, auto-generating Apache Beam API code from Calcite logical plan.

Please let us know if you guys have comments and suggestions on the contribution. I will create a pull request soon.

Thanks,
Khai