You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@calcite.apache.org by "Bar David, Shachar" <Sh...@intuit.com.INVALID> on 2020/10/16 05:30:27 UTC

Using Spark and Calcite to find the column lineage and rewrite spark queries

Hello,
I'm using Calcite to:

  1.  Extract the column lineage of a Spark SQL query
  2.  Rewrite SQL queries, specifically select statements to one or more merge statements

I was able to accomplish 1 and some of 2 however, currently I'm using the file adapter and some CSV files
given that I was unable to connect Calcite to my SparkContext, such that the spark UDFs and tables are available to Calcite allowing me to parse the queries,

Is there any way to accomplish that? should I be using Avatica?


Regards,
Shachar Bar David,
Group Manager, Intuit Curation Platform.

zoom: https://intuit.zoom.us/j/6405986798