DataFrame Vs RDDs ... Which one to use When ?

[Divya Gehlot <di...@gmail.com>]

Hi,
I am new bee to spark and a bit confused about RDDs and DataFames in Spark.
Can somebody explain me with the use cases which one to use when ?

Would really appreciate the clarification .

Thanks,
Divya

Re: DataFrame Vs RDDs ... Which one to use When ?

[Chris Fregly <ch...@fregly.com>]

here's a good article that sums it up, in my opinion:

https://ogirardot.wordpress.com/2015/05/29/rdds-are-the-new-bytecode-of-apache-spark/

basically, building apps with RDDs is like building with apps with
primitive JVM bytecode.  haha.

@richard:  remember that even if you're currently writing RDDs in
Java/Scala, you're not gaining the code gen/rewrite performance benefits of
the Catalyst optimizer.

i agree with @daniel who suggested that you start with DataFrames and
revert to RDDs only when DataFrames don't give you what you need.

the only time i use RDDs directly these days is when i'm dealing with a
Spark library that has not yet moved to DataFrames - ie. GraphX - and it's
kind of annoying switching back and forth.

almost everything you need should be in the DataFrame API.

Datasets are similar to RDDs, but give you strong compile-time typing,
tabular structure, and Catalyst optimizations.

hopefully Datasets is the last API we see from Spark SQL...  i'm getting
tired of re-writing slides and book chapters!  :)

On Mon, Dec 28, 2015 at 4:55 PM, Richard Eggert <ri...@gmail.com>
wrote:

> One advantage of RDD's over DataFrames is that RDD's allow you to use your
> own data types, whereas DataFrames are backed by RDD's of Record objects,
> which are pretty flexible but don't give you much in the way of
> compile-time type checking. If you have an RDD of case class elements or
> JSON, then Spark SQL can automatically figure out how to convert it into an
> RDD of Record objects (and therefore a DataFrame), but there's no way to
> automatically go the other way (from DataFrame/Record back to custom types).
>
> In general, you can ultimately do more with RDDs than DataFrames, but
> DataFrames give you a lot of niceties (automatic query optimization, table
> joins, SQL-like syntax, etc.) for free, and can avoid some of the runtime
> overhead associated with writing RDD code in a non-JVM language (such as
> Python or R), since the query optimizer is effectively creating the
> required JVM code under the hood. There's little to no performance benefit
> if you're already writing Java or Scala code, however (and RDD-based code
> may actually perform better in some cases, if you're willing to carefully
> tune your code).
>
> On Mon, Dec 28, 2015 at 3:05 PM, Daniel Siegmann <
> daniel.siegmann@teamaol.com> wrote:
>
>> DataFrames are a higher level API for working with tabular data - RDDs
>> are used underneath. You can use either and easily convert between them in
>> your code as necessary.
>>
>> DataFrames provide a nice abstraction for many cases, so it may be easier
>> to code against them. Though if you're used to thinking in terms of
>> collections rather than tables, you may find RDDs more natural. Data frames
>> can also be faster, since Spark will do some optimizations under the hood -
>> if you are using PySpark, this will avoid the overhead. Data frames may
>> also perform better if you're reading structured data, such as a Hive table
>> or Parquet files.
>>
>> I recommend you prefer data frames, switching over to RDDs as necessary
>> (when you need to perform an operation not supported by data frames / Spark
>> SQL).
>>
>> HOWEVER (and this is a big one), Spark 1.6 will have yet another API -
>> datasets. The release of Spark 1.6 is currently being finalized and I would
>> expect it in the next few days. You will probably want to use the new API
>> once it's available.
>>
>>
>> On Sun, Dec 27, 2015 at 9:18 PM, Divya Gehlot <di...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I am new bee to spark and a bit confused about RDDs and DataFames in
>>> Spark.
>>> Can somebody explain me with the use cases which one to use when ?
>>>
>>> Would really appreciate the clarification .
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>
>
> --
> Rich
>



-- 

*Chris Fregly*
Principal Data Solutions Engineer
IBM Spark Technology Center, San Francisco, CA
http://spark.tc | http://advancedspark.com

Re: DataFrame Vs RDDs ... Which one to use When ?

[Richard Eggert <ri...@gmail.com>]

One advantage of RDD's over DataFrames is that RDD's allow you to use your
own data types, whereas DataFrames are backed by RDD's of Record objects,
which are pretty flexible but don't give you much in the way of
compile-time type checking. If you have an RDD of case class elements or
JSON, then Spark SQL can automatically figure out how to convert it into an
RDD of Record objects (and therefore a DataFrame), but there's no way to
automatically go the other way (from DataFrame/Record back to custom types).

In general, you can ultimately do more with RDDs than DataFrames, but
DataFrames give you a lot of niceties (automatic query optimization, table
joins, SQL-like syntax, etc.) for free, and can avoid some of the runtime
overhead associated with writing RDD code in a non-JVM language (such as
Python or R), since the query optimizer is effectively creating the
required JVM code under the hood. There's little to no performance benefit
if you're already writing Java or Scala code, however (and RDD-based code
may actually perform better in some cases, if you're willing to carefully
tune your code).

On Mon, Dec 28, 2015 at 3:05 PM, Daniel Siegmann <
daniel.siegmann@teamaol.com> wrote:

> DataFrames are a higher level API for working with tabular data - RDDs are
> used underneath. You can use either and easily convert between them in your
> code as necessary.
>
> DataFrames provide a nice abstraction for many cases, so it may be easier
> to code against them. Though if you're used to thinking in terms of
> collections rather than tables, you may find RDDs more natural. Data frames
> can also be faster, since Spark will do some optimizations under the hood -
> if you are using PySpark, this will avoid the overhead. Data frames may
> also perform better if you're reading structured data, such as a Hive table
> or Parquet files.
>
> I recommend you prefer data frames, switching over to RDDs as necessary
> (when you need to perform an operation not supported by data frames / Spark
> SQL).
>
> HOWEVER (and this is a big one), Spark 1.6 will have yet another API -
> datasets. The release of Spark 1.6 is currently being finalized and I would
> expect it in the next few days. You will probably want to use the new API
> once it's available.
>
>
> On Sun, Dec 27, 2015 at 9:18 PM, Divya Gehlot <di...@gmail.com>
> wrote:
>
>> Hi,
>> I am new bee to spark and a bit confused about RDDs and DataFames in
>> Spark.
>> Can somebody explain me with the use cases which one to use when ?
>>
>> Would really appreciate the clarification .
>>
>> Thanks,
>> Divya
>>
>
>


-- 
Rich

Re: DataFrame Vs RDDs ... Which one to use When ?

[Daniel Siegmann <da...@teamaol.com>]

DataFrames are a higher level API for working with tabular data - RDDs are
used underneath. You can use either and easily convert between them in your
code as necessary.

DataFrames provide a nice abstraction for many cases, so it may be easier
to code against them. Though if you're used to thinking in terms of
collections rather than tables, you may find RDDs more natural. Data frames
can also be faster, since Spark will do some optimizations under the hood -
if you are using PySpark, this will avoid the overhead. Data frames may
also perform better if you're reading structured data, such as a Hive table
or Parquet files.

I recommend you prefer data frames, switching over to RDDs as necessary
(when you need to perform an operation not supported by data frames / Spark
SQL).

HOWEVER (and this is a big one), Spark 1.6 will have yet another API -
datasets. The release of Spark 1.6 is currently being finalized and I would
expect it in the next few days. You will probably want to use the new API
once it's available.


On Sun, Dec 27, 2015 at 9:18 PM, Divya Gehlot <di...@gmail.com>
wrote:

> Hi,
> I am new bee to spark and a bit confused about RDDs and DataFames in Spark.
> Can somebody explain me with the use cases which one to use when ?
>
> Would really appreciate the clarification .
>
> Thanks,
> Divya
>