You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@calcite.apache.org by "Wangfei (X)" <wa...@huawei.com> on 2016/09/02 03:35:00 UTC

how about integrate spark dataset/dataframe api

Hi, community
       I noticed that now the spark adapter in calcite is integrated 
with spark core api, since now the dataset/dataframe api become the top 
level api, how about integrate the dataset/dataframe api ? or is it 
possible to do that?

Fei.

Re: how about integrate spark dataset/dataframe api

Posted by Julian Hyde <jh...@apache.org>.

It’s an interesting idea. I know that the data frame API is easier to work with for application developers, but since Calcite would be generating the code, can you describe the benefits to the Calcite user of changing the integration point?

It’s definitely true that Calcite’s Spark adapter needs some love. If someone would like to rework the adapter in terms of the data frame API and get it working on more cases, and more reliably, I would definitely welcome it.

Julian

> On Sep 1, 2016, at 8:35 PM, Wangfei (X) <wa...@huawei.com> wrote:
> 
> Hi, community
>      I noticed that now the spark adapter in calcite is integrated with spark core api, since now the dataset/dataframe api become the top level api, how about integrate the dataset/dataframe api ? or is it possible to do that?
> 
> Fei.
>