You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Divya Gehlot <di...@gmail.com> on 2016/05/10 02:36:44 UTC

best fit - Dataframe and spark sql use cases

Hi,
I would like to know the uses cases where data frames is best fit and use
cases where Spark SQL is best fit based on the one's  experience .


Thanks,
Divya

Re: best fit - Dataframe and spark sql use cases

Posted by Mathieu Longtin <ma...@closetwork.org>.

Spark SQL is translated to DataFrame operations by the SQL engine. Use
whichever is more comfortable for the task. Unless I'm doing something very
straight forward, I go with SQL, since any improvement to the SQL engine
will improve the resulting DataFrame operations. Hard-coded DataFrame
operation won't change even if a better operation becomes available.

On Mon, May 9, 2016 at 10:37 PM Divya Gehlot <di...@gmail.com>
wrote:

> Hi,
> I would like to know the uses cases where data frames is best fit and use
> cases where Spark SQL is best fit based on the one's  experience .
>
>
> Thanks,
> Divya
>
>
>
>
>
> --
Mathieu Longtin
1-514-803-8977