You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@calcite.apache.org by Bertty Contreras <be...@scalytics.io> on 2020/09/30 23:00:07 UTC

Code of Spark in the Core

Hi Guys,

I'm thinking of using calcite on my project, but I found something that
bothers me about the incongruity from the documentation and some pieces of
the code .

you said on the documentation "Calcite intentionally stays out of the
business of storing and
processing data"(https://calcite.apache.org/docs/index.html), but when i
was analyzing the code of the core I found one part (links below) where you
use SparkHandler or SparkContext.

The question is, are you thinking to use spark in the near future as a
processing engine or that code needs to be there because of the adapter of
spark? If the answer is yes to the second question, why is it not on the
Spark adapter instead of the core ?

Thank you so much for your time,

Best regards


https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L116-L117

https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L132-L150

https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L152-L233

https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L173

Re: Code of Spark in the Core

Posted by Julian Hyde <jh...@gmail.com>.
The SparkContext is a bit embarrassing. Most adapters are able to create a connection based on purely textual credentials, but the Spark adapter has to start off from an object. So, SparkContext is part of the Calcite instance.

It could perhaps be cleaned up, but on the other hand it is not doing us very much harm.

We are not ’thinking to use Spark as a processing engine’. Effectively, we already do - but only if you use the Spark adapter.

Julian


> On Sep 30, 2020, at 4:00 PM, Bertty Contreras <be...@scalytics.io> wrote:
> 
> Hi Guys,
> 
> I'm thinking of using calcite on my project, but I found something that
> bothers me about the incongruity from the documentation and some pieces of
> the code .
> 
> you said on the documentation "Calcite intentionally stays out of the
> business of storing and
> processing data"(https://calcite.apache.org/docs/index.html), but when i
> was analyzing the code of the core I found one part (links below) where you
> use SparkHandler or SparkContext.
> 
> The question is, are you thinking to use spark in the near future as a
> processing engine or that code needs to be there because of the adapter of
> spark? If the answer is yes to the second question, why is it not on the
> Spark adapter instead of the core ?
> 
> Thank you so much for your time,
> 
> Best regards
> 
> 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L116-L117
> 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L132-L150
> 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L152-L233
> 
> https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/jdbc/CalcitePrepare.java#L173