Execute Calcite logical plans on Spark

[Khai Tran <kh...@linkedin.com.INVALID>]

Hi all,
I'm trying to figure out some way to execute a given Calcite logical plan
on Spark. One possible way to convert the Calcite plan back to SQL and run
it as Spark SQL.

Any ideas on that?

Thanks,
Khai

Re: Execute Calcite logical plans on Spark

[Julian Hyde <jh...@apache.org>]

Since the SQL will pass through Spark’s Catalyst SQL optimizer, I suspect that most of the optimizations Calcite has made will be negated.

The ideal thing would be to translate the Calcite plan (RelNodes) into a Spark program, which would then be executed without further optimization. Calcite’s Spark adapter does that (but is old and incomplete); I heard rumors of an improved Spark adapter but it hasn’t landed in open source yet, and I don’t know when/whether that will happen.

Julian


> On Apr 21, 2017, at 3:50 PM, Khai Tran <kh...@linkedin.com.INVALID> wrote:
> 
> Hi all,
> I'm trying to figure out some way to execute a given Calcite logical plan
> on Spark. One possible way to convert the Calcite plan back to SQL and run
> it as Spark SQL.
> 
> Any ideas on that?
> 
> Thanks,
> Khai